Group Management Reference¶
Group management provides consistent list of active DGI processes as well as identifies a primary process for initiating distributed algorithms.
Using Group Management¶
GMAgent::ProcessPeerList is available by including gm/GroupManagement.hpp
Group management will send all modules in the system a PeerList message when the list of active processes in the system changes. This message will also contain the identity of the leader process. Group management provides a static method for to aid processing this message.
-
PeerSet
ProcessPeerList
(const PeerListMessage & msg) Handles Processing a PeerList.
- Description:
- Provides a utility function for correctly handling incoming peer lists.
- Return
- A PeerSet with all nodes in the group.
- Parameters
msg
-The message to parse
This method returns a PeerList of the processes in the group. The sender of the message will always be the group leader.
An example of processing the PeerList message is in Receiving Messages
Physical Topology¶
Group management can be configured to respect the topology of the physical network: it will not group two DGI unless a physical path exists between the two processes (as opposed to a cyber path).
Directions on configuring physical topology can be found in Physical Topology
Embedding Group State In Simulation¶
The group state is by manipulating a “Logger” device on the FREEDM system. The value of device is updated each time the check or timeout procedure is called, that is, once at the beginning of the Group Management phase.
Data is stored in the device as a bit field. Because of the way data is passed to and from the RTDS and PSCAD the data is transported as a float, although the individual bits are unaffected by this process, it may be necessary to convert the float to an unsigned integer to be able to preform the bit fiddling needed to access the information.
This setup assumes that all DGI are aware of all other DGI in the system and that all the Group Management status tables are the same. That is, there is not a DGI some DGI is aware of that some other DGI isn’t aware of. This is reasonable because of the experiments we are currently running, and the fact that the container for the other DGIs in the system is a map so when iterated the items are returned in alpha numeric order.
UUIDS ARE ENTERED IN THE BITFIELD IN ASCENDING ALPHANUMERIC ORDER.
## Bit Field Structure
Note that 2^0 is set based on the endianess of an x86_64 machine.
- (2^0) - Set to 1 if the DGI editing the device is the coordinator.
- (2^1) - Set to 1 if the DGI with the first UUID is in the same group as the DGI editing the bitfield
- (2^2) - Set to 1 if the DGI with the second UUID is in the same group as the DGI editing the bitfield.
- (2^n) - Set to 1 if the DGI with the nth UUID is in the same group as the DGI editing the bitfield.
Bits of non-members will be set to 0. The SGI will always set the bit relating to its own UUID to 1.
In order to use this feature, you need to define a Logger device with a “groupStatus” field. The DGI will write its group state to the first Logger device on the system.
Implementation Details¶
Algorithm¶
Group Management is an implementation of the Garcia-Molina Invitation Election Algorithm found in “Elections in a Distributed System” published 1982 in IEEE Transactions on Computers.
GMAgent Reference¶
- class
Declaration of Garcia-Molina Invitation Leader Election algorithm.
Public Types
- enum [anonymous]
Module states.
Values:
-
NORMAL
-
DOWN
-
RECOVERY
-
REORGANIZATION
-
ELECTION
-
Public Functions
-
GMAgent
() Constructor for using this object as a module.
GMAgent::GMAgent
- Description:
- Constructor for the group management module.
- Limitations:
- None
- Precondition:
- None
- Postcondition:
- Object initialized and ready to enter run state.
-
~GMAgent
() Module destructor.
GMAgent::~GMAgent
- Description:
- Class desctructor
- Precondition:
- None
- Postcondition:
- The object is ready to be destroyed.
-
int
Run
() Called to start the system.
GMAgent::Run
- Description:
- Main function which initiates the algorithm
- Precondition:
- connections to peers should be instantiated
- Postcondition:
- execution of group management algorithm
Public Static Functions
-
PeerSet
ProcessPeerList
(const PeerListMessage & msg) Handles Processing a PeerList.
- Description:
- Provides a utility function for correctly handling incoming peer lists.
- Return
- A PeerSet with all nodes in the group.
- Parameters
msg
-The message to parse
Protected Functions
-
CPeerNode
GetMe
() Gets a CPeerNode representing this process.
GetMe
- Description:
- Gets a CPeerNode that refers to this process.
- Return
- A CPeerNode referring to this process.
-
std::string
GetUUID
() const Gets the UUID of this process.
GetUUID
- Description:
- Gets this process’s UUID.
- Return
- This process’s UUID
Private Functions
-
ModuleMessage
Accept
() Creates an Accept Message.
GMAgent::Accept
- Description:
- Creates a new accept message from this node
- Precondition:
- This node is in a group.
- Postcondition:
- No change.
- Return
- A GroupManagementMessage with the contents of an Accept message
-
CPeerNode
AddPeer
(std::string uuid) Adds a peer to the peer set from UUID.
GMAgent::AddPeer
- Description:
- Adds a peer to allpeers by uuid.
- Precondition:
- the UUID is registered in the connection manager
- Postcondition:
- A new peer object is created and inserted in CGlobalPeerList::instance().
- Return
- A pointer to the new peer
- Parameters
uuid
-of the peer to add.
-
CPeerNode
AddPeer
(CPeerNode peer) Adds a peer from a pointer to a peer node object.
GMAgent::AddPeer
- Description:
- Adds a peer to all peers by pointer.
- Precondition:
- the pointer is valid
- Postcondition:
- The peer object is inserted in CGlobalPeerList::instance().
- Return
- A pointer that is the same as the input pointer
- Parameters
peer
-a pointer to a peer.
-
ModuleMessage
AreYouCoordinator
() Creates AYC Message.
GMAgent::AreYouCoordinator
- Description:
- Creates a new Are You Coordinator message, from this object
- Precondition:
- The UUID is set
- Postcondition:
- No change
- Return
- A GroupManagementMessage with the contents of an Are You Coordinator Message.
- Limitations:
- : Can only author messages from this node.
-
ModuleMessage
AreYouCoordinatorResponse
(std::string payload, int seq) Creates A Response message.
GMAgent::AreYouCoordinatorResponse
- Description:
- Creates a response message (Yes/No) message from this node
- Precondition:
- This node has a UUID.
- Postcondition:
- No change.
- Return
- A GroupManagementMessage with the contents of a Response message
- Parameters
payload
-Response message (typically yes or no)
seq
-sequence number? (?)
-
ModuleMessage
AreYouThere
() Creates a AYT, used for Timeout.
GMAgent::AreYouThere
- Description:
- Creates a new AreYouThere message from this node
- Precondition:
- This node is in a group.
- Postcondition:
- No Change.
- Return
- A GroupManagementMessage with the contents of an AreYouThere message
-
ModuleMessage
AreYouThereResponse
(std::string payload, int seq) Creates A Response message.
GMAgent::AreYouThereResponse
- Description:
- Creates a response message (Yes/No) message from this node
- Precondition:
- This node has a UUID.
- Postcondition:
- No change.
- Return
- A GroupManagementMessage with the contents of a Response message
- Parameters
payload
-Response message (typically yes or no)
seq
-sequence number? (?)
-
void
Check
(const boost::system::error_code & err) Checks for other up leaders.
GMAgent::Check
- Description:
- This method queries all nodes to Check and see if any of them consider themselves to be Coordinators.
- Precondition:
- This node is in the normal state.
- Postcondition:
- Output of a system state and sent messages to all nodes to Check for Coordinators.
- Citation:
- GroupManagement (Check).
- Parameters
err
-Error associated with calling timer.
-
std::string
Coordinator
() const Returns the coordinators uuid.
-
CPeerNode
GetPeer
(const std::string & uuid) Gets a pointer to a peer from UUID.
GMAgent::GetPeer
- Description:
- Gets a peer from All Peers by uuid
- Precondition:
- None
- Postcondition:
- None
- Return
- A pointer to the requested peer, or a null pointer if the peer could not be found.
- Parameters
uuid
-The uuid of the peer to fetch
-
int
GetStatus
() const Gets the status of a node.
- Description:
- returns the status stored in the node as an an integer. This means that in an inherited class, you may either define integer constants or an enumeration to generate status values.
-
void
HandleAccept
(const AcceptMessage & msg, CPeerNode peer) Handles recieving accept messsages.
GMAgent::HandleAccept
- Description:
- Handles receiveing the invite accept
- Key:
- gm.Accept
- Precondition:
- In an election, and invites have been sent out.
- Postcondition:
- The sender is added to the tenative list of accepted peers for the group, if the accept messag is for the correct group identifier.
- Interaction Peers:
- Any node which has received an invite. (This can be any selection of the global peerlist, not just the ones this specific node sent the message to.)
-
void
HandleAreYouCoordinator
(const AreYouCoordinatorMessage & msg, CPeerNode peer) Handles recieving are you coordinator messages.
GMAgent::HandleAreYouCoordinator
- Description:
- Handles receiveing the AYC message
- Key:
- gm.AreYouCoordinator
- Precondition:
- None
- Postcondition:
- The node responds yes or no to the request.
- Interaction Peers:
- Any node in the system is eligible to receive this at any time.
-
void
HandleAreYouThere
(const AreYouThereMessage & msg, CPeerNode peer) Handles recieving are you there messsages.
GMAgent::HandleAreYouThere
- Description:
- Handles recieving the AYT message
- Key:
- gm.AreYouThere
- Precondition:
- None
- Postcondition:
- The node responds yes or no to the request.
- Interaction Peers:
- Any node in the system is eligible to receive this message at any time.
-
virtual void
HandleIncomingMessage
(boost::shared_ptr< const ModuleMessage > msg, CPeerNode peer) Handles received messages.
“Downcasts” incoming messages into a specific message type, and passes the message to an appropriate handler.
- Parameters
msg
-the incoming message
peer
-the node that sent this message (could be this DGI)
-
void
HandleInvite
(const InviteMessage & msg, CPeerNode peer) Handles recieving invite messages.
GMAgent::HandleInvite
- Description:
- Handles recieving the invite
- Key:
- gm.Invite
- Precondition:
- The system is in the NORMAL state
- Postcondition:
- The system updates its group to accept the invite and switches to reorganization mode; it will wait for a timeout. If the Ready/Peerlist has not arrived, it will enter recovery. Otherwise, it will resume work as part of the group it was invited to.
- Interaction Peers:
- Any node could send an invite at any time.
-
void
HandlePeerList
(const PeerListMessage & msg, CPeerNode peer) Hadles recieving peerlists.
GMAgent::HandlePeerList
- Description:
- Handles receiveing the peerlist.
- Key:
- any.PeerList
- Precondition:
- The node is in the reorganization or normal state
- Postcondition:
- The node’s peerlist is updated to match that of the incoming message if the message has come from his coordinator.
- Interaction Peers:
- Coordinator only.
-
void
HandlePeerListQuery
(const PeerListQueryMessage & msg, CPeerNode peer) Handles recieving peerlist requests.
GMAgent::HandlePeerListQuery
- Description:
- Handles responding to peerlist queries.
- Key:
- gm.PeerListQuery
- Precondition:
- None
- Postcondition:
- Dispatched a message to the requester with a peerlist.
- Interaction Peers:
- Any. (Local or remote, doesn’t have to be a member of group)
-
void
HandleResponseAYC
(const AreYouCoordinatorResponseMessage & msg, CPeerNode peer) Handles recieving AYC responses.
GMAgent::HandleResponseAYC
- Description:
- Handles recieving the Response
- Key:
- gm.Response.AreYouCoordinator
- Precondition:
- The timer for the AYC request batch the original request was a part of has not expired
- Postcondition:
- The response from the sender is noted, if yes, the groups will attempt to merge in the future.
- Interaction Peers:
- A node the AYC request was sent to.
-
void
HandleResponseAYT
(const AreYouThereResponseMessage & msg, CPeerNode peere) Handles recieving AYT responses.
GMAgent::HandleResponseAYT
- Description:
- Handles recieving the Response
- Key:
- gm.Response.AreYouThere
- Precondition:
- The timer for the AYT request batch (typically 1) the original request was a part of has not expired
- Postcondition:
- The response from the sender is noted. If the answer is no, then this node will enter recovery, if the recovery timer has not already been set, otherwise we will allow it to expire.
-
ModuleMessage
Invitation
() Creates Group Invitation Message.
GMAgent::Invitation
- Description:
- Creates a new invation message from the leader of this node’s current leader, to join this group.
- Precondition:
- The node is currently in a group.
- Postcondition:
- No change
- Return
- A GroupManagementMessage with the contents of a Invitation message.
-
void
InviteGroupNodes
(const boost::system::error_code & err, PeerSet p_tempSet) Sends invitations to all group members.
GMAgent::InviteGroupNodes
- Description:
- This function will invite all the members of a node’s old group to join it’s new group.
- Precondition:
- None
- Postcondition:
- Invitations have been sent to members of this node’s old group. if this node is a Coordinator, this node sets a timer for Reorganize
- Citation:
- Group Management (Merge)
- Parameters
err
-The error message associated with the calling thimer
p_tempSet
-The set of nodes that were members of the old group.
-
bool
IsCoordinator
() const Returns true if this node considers itself a coordinator.
-
void
Merge
(const boost::system::error_code & err) Sends invitations to all known nodes.
GMAgent::Merge
- Description:
- If this node is a Coordinator, this method sends invites to join this node’s group to all Coordinators, and then makes a call to second function to invite all current members of this node’s old group to the new group.
- Precondition:
- This node has waited for a Premerge.
- Postcondition:
- If this node was a Coordinator, this node has been placed in an election state. Additionally, invitations have been sent to all Coordinators this node is aware of. Lastly, this function has called a function or set a timer to invite its old group nodes.
- Citation:
- Group Management (Merge)
- Parameters
err
-A error associated with the calling timer.
-
ModuleMessage
PeerList
(std::string requester = "all") Generates a peer list.
GMAgent::PeerList
- Description:
- Packs the group list (Up_Nodes) in GroupManagementMessage
- Precondition:
- This node is a leader.
- Postcondition:
- No Change.
- Return
- A GroupManagementMessage with the contents of group membership
-
void
Premerge
(const boost::system::error_code & err) Waits a time period determined by UUID for merge.
GMAgent::Premerge
- Description:
- Handles a proportional wait prior to calling Merge
- Precondition:
- Check has been called and responses have been collected from other nodes. This node is a Coordinator.
- Postcondition:
- A timer has been set based on this node’s UUID to break up ties before merging.
- Citation:
- GroupManagement (Merge)
- Parameters
err
-An error associated with the clling timer.
-
void
PushPeerList
() Sends the peer list to all group members.
GMAgent::PushPeerList
- Description:
- Sends the membership list to other modules of this node and other nodes
- Precondition:
- This node is new group leader
- Postcondition:
- A peer list is pushed to the group members
- Return
- Nothing
-
void
Recovery
() Resets the algorithm to the default startup state.
GMAgent::Recovery
- Description:
- The method used to set or reset a node into a “solo” state where it is its own leader. To do this, it forms an empty group, then enters a normal state.
- Precondition:
- None
- Postcondition:
- The node enters a NORMAL state.
- Citation:
- Group Management Algorithmn (Recovery).
-
void
Recovery
(const boost::system::error_code & err) Handles no response from timeout message.
GMAgent::Recovery
- Description:
- Recovery function extension for handling timer expirations.
- Precondition:
- Some timer leading to this function has expired or been canceled.
- Postcondition:
- If the timer has expired, Recovery begins, if the timer was canceled and this node is NOT a Coordinator, this will cause a Timeout Check using Timeout()
- Parameters
err
-The error code associated with the calling timer.
-
void
Reorganize
(const boost::system::error_code & err) Puts the system into the working state.
GMAgent::Reorganize
- Description:
- Organizes the members of the group and prepares them to do their much needed work!
- Precondition:
- The node is a leader of group.
- Postcondition:
- The group has received work assignments, ready messages have been sent out, and this node enters the NORMAL state.
- Citation:
- Group Management Reorganize
-
void
SetStatus
(int status) Sets the status of the node.
- Description:
- Sets the internal status variable, m_status based on the paramter status. See GetStatus for tips on establishing the status code numbers.
- Precondition:
- None
- Postcondition:
- The modules status code has been set to “status”
- Parameters
status
-The status code to set
-
void
StartMonitor
(const boost::system::error_code & err) Start the monitor after transient is over.
-
void
SystemState
() Outputs information about the current state to the logger.
GMAgent::SystemState
- Description:
- Puts the system state to the logger.
- Precondition:
- None
- Postcondition:
- None
-
void
Timeout
(const boost::system::error_code & err) Checks that the leader is still alive and working.
GMAgent::Timeout
- Description:
- Sends an AreYouThere message to the coordinator and sets a timer if the timer expires, this node goes into recovery.
- Precondition:
- The node is a member of a group, but not the leader
- Postcondition:
- A timer for recovery is set.
- Citation:
- Group Managment Timeout
Private Members
-
boost::posix_time::time_duration
AYC_RESPONSE_TIMEOUT
How long to wait for responses from other nodes.
-
boost::posix_time::time_duration
AYT_RESPONSE_TIMEOUT
How long to wait for responses from other nodes.
-
boost::posix_time::time_duration
CHECK_TIMEOUT
How long between AYC checks.
-
boost::posix_time::time_duration
FID_TIMEOUT
How long to wait before checking attached FIDs.
-
boost::posix_time::time_duration
INVITE_RESPONSE_TIMEOUT
How long to wait for responses from other nodes.
-
TimedPeerSet
m_AYCResponse
Nodes expecting AYC response from.
-
TimedPeerSet
m_AYTResponse
Nodes expecting AYT response from.
-
PeerSet
m_Coordinators
Known Coordinators.
-
std::map< std::string, bool >
m_fidstate
A store for the state of attached FIDs.
-
CBroker::TimerHandle
m_fidtimer
Timer for checking FIDs.
-
google::protobuf::uint32
m_GroupID
The ID number of the current group (Never initialized for fun)
-
std::string
m_GroupLeader
The uuid of the group leader.
-
int
m_groupsbroken
Number of groups broken.
-
int
m_groupselection
Number of elections started.
-
int
m_groupsformed
Number of groups formed.
-
int
m_groupsjoined
Number of accepts sent.
-
unsigned int
m_GrpCounter
The number of groups being formed.
-
boost::asio::io_service
m_localservice
The io_service used.
-
int
m_membership
Total size of groups after all checks.
-
int
m_membershipchecks
Number of membership checks.
-
int
m_status
A store for the status of this node.
-
CBroker::TimerHandle
m_timer
A timer for stepping through the election process.
-
PeerSet
m_UpNodes
Nodes In My Group.
-
boost::posix_time::time_duration
TIMEOUT_TIMEOUT
How long beteween AYT checks.
Private Static Functions
-
ModuleMessage
PeerListQuery
(std::string requester) Generates a CMessage that can be used to query for the group.
GMAgent::PeerListQuery
- Description:
- Generates a GroupManagementMessage that can be used to query the peerlist of a node.
- Precondition:
- : None
- Postcondition:
- : No change
- Return
- A GroupManagementMessage which can be used to query for
- Parameters
requester
-The module who the response should be addressed to.
-
ModuleMessage
PrepareForSending
(const GroupManagementMessage & message, std::string recipient = "gm") Wraps a GroupManagementMessage in a ModuleMessage.
Wraps a GroupManagementMessage in a ModuleMessage.
- Return
- a ModuleMessage containing a copy of the GroupManagementMessage
- Parameters
message
-the message to prepare. If any required field is unset, the DGI will abort.
recipient
-the module (sc/lb/gm/clk etc.) the message should be delivered to