OpenSAF Development Process

1 Introduction

This document outlines the process for the OpenSAF development model. This document is intended to be read by everybody who intends to join and contribute to the OpenSAF project. In particular, it describes how the Membership at Large, the Board of Directors, and the OpenSAF Technical Leadership Council (TLC) lead, influence, and collaborate to achieve the purposes of the OpenSAF Foundation (described in section 1.4 of the OpenSAF bylaws). This document has 2 principal sections:

• OpenSAF Development Process Model
• OpenSAF Rules of Engagement

The OpenSAF development process is designed to be democratic. Appropriate voting points are built into the process to allow for resolution of contentious issues. All votes specified in this document are run according to the voting rules specified in the OpenSAF Foundation's Technical Operating Procedures.

2 OpenSAF Development Process Model

OpenSAF encourages active community participation and believes in an open, transparent and fair development process that maximizes community involvement and participation. However to ensure that the project always produces high quality code it requires that all contributions made to OpenSAF be carefully scrutinized and reviewed before acceptance. This is the single most important factor guiding the development process for OpenSAF that may be divided up into the following distinct phases. In each of these phases the responsibilities of the “Roles” described in the  Technical Operating Procedures is depicted.

2.1 Defining the Project Roadmap

OpenSAF uses roadmaps to identify and prioritize work that needs to be done. The Roadmap is prepared and maintained by the TLC. Please refer to Section 3 for a detailed description of how OpenSAF roadmaps are built as well as how new features show up in the OpenSAF roadmap. The Technical co-chairs are responsible for keeping the OpenSAF Foundation Board of Directors well informed of all key decision affecting the Roadmap and schedules of the OpenSAF project(s). The Roadmap must be consistent with the Purposes as described in OpenSAF Foundation Bylaws section 1.4.

2.2 Roadmap Definition Process

The following process shall be used to create or update the OpenSAF Roadmap.

1. Users and Developers shall use the "rules of engagement", described in section 3 of this document, to submit proposals for OpenSAF change requests.
2. Once a quarter, the TLC shall review accumulated change request proposals, as defined in section 3 of this document, and render approvals or rejections as appropriate.
3. All approved change requests shall become potential candidates for inclusion on the OpenSAF Roadmap.
4. In order for an approved change request to be placed on the roadmap, the request owners shall provide the TLC with implementation estimates. A given change request shall not included in the roadmap unless the implementation estimates are provided within the time limit allotted by the TLC.
5. A draft of the Roadmap is produced. The order of change requests on the Roadmap shall be determined **first** by the importance, based on the overall strategic and technical direction established by the OpenSAF Foundation Vision Statement, and **second** by implementation estimates.
6. A draft Roadmap or draft Roadmap update shall be distributed to the OpenSAF membership and user community at large for comment and feedback in advance of its adoption. This distribution and all discussion and debate around the Roadmap shall be held in an open and transparent public a public forum, such as a mailing list or newsgroup. Prior to any TLC vote to approve a Roadmap or Roadmap update, every member and user of the OpenSAF community shall have the right to communicate concerns and objections to the TLC.
7. Once a draft Roadmap, or draft Roadmap update, has been ratified by the OpenSAF membership, is it sent to the OpenSAF BoD for advice and consent.
8. Draft Roadmap, or draft Roadmap update, becomes official when jointly ratified by the members of the BoD and TLC.

Additionally the users, developers and committers use the bug tracker and feature enhancement request databases to report bugs and request enhancements. Such requests made by the community are considered while prioritizing roadmaps for the future OpenSAF releases.

Responsibilities:

• TLC - Develops and finalizes the feature roadmap with the advice and consent of the BoD. The Roadmap is voted on by the TLC before it is passed on to the BoD for approval.
• Committers, Developers and Users - Influence the roadmap by submitting bugs and feature enhancement requests. Advise the TLC of critical issues which need to be addressed in the roadmap.

2.3 Distribution & Co-ordination of Work

The Technical Leadership Council distributes and coordinates work amongst interested developers. Usually developers or a group of developers volunteer to take on an item from the OpenSAF roadmap for a future release. In case a feature on the roadmap has no volunteers the Technical Leadership Community refers this to the BoD for appropriate action such as removal or postponement of the feature from the roadmap. Additionally OpenSAF developers may provide patches and solutions to OpenSAF problem reports.

Responsibilities:

• TLC - Coordinates and distributes work amongst developers. It ensures that development is collaborative and not competitive.
• Committers and Developers - Volunteer to work on certain features. Announce their involvement in a particular area and invite other people to join.

2.4 Architecture and Design Review

Major feature extensions like implementation of a new SA Forum Service or a change that affects multiple modules of OpenSAF undergoes a design review phase by the Technical Leadership Council. This step is not mandatory for all features. Anyone planning to contribute such features needs to submit their design of the feature to the TLC for approval.

The TLC categorizes the items on the OpenSAF roadmap that need this Architecture/Design? Review phase. Certain other bugs submitted for the OpenSAF project may be classified as critical by the TLC which also undergoes this review phase. The TLC designates a group of individuals to review and approve the architecture and design of the proposed feature.

Responsibilities:

• TLC - Designate a review team that undertakes the architecture and design review for critical features. The review team will be usually composed of domain experts in the respective areas.
• Committers, Developers & Users - The Review team that the TLC appoints usually gets chosen from the Committers, Developers and Users. All OpenSAF participants are free to critique architecture & design submissions in addition to the designated reviewers.

2.4.1 Voting Points

1. If there is contention about whether a feature is complicated enough to warrant a design review phase, a TLC vote can be held on the matter.
2. If there is contention about whether a feature has adequately completed the architecture/design review phase, a TLC vote can be held on the matter.

2.5 Pre-Check-In Testing

It is expected that OpenSAF developers run regression and smoke tests that are provided by the OpenSAF project before announcing a feature on the OpenSAF mailing lists as complete and ready to be reviewed. Such testing includes tests that ensure backward compatibility and in-service upgradeability of the OpenSAF code base. It is also expected that OpenSAF developers do sufficient unit testing of their patches & features before advertising such features or patches as basic-review-ready. There’s no way that this step can be enforced but a broken tree would speak for itself.

It is possible that the feature or the patch that a developer contributes to the project is new and no existing tests can verify such a modification to the OpenSAF code base. In such cases the developer is expected to contribute test suites into the OpenSAF test harness that verifies the specific modification in question.

Responsibilities:

• Developers: - Verify and Test their contribution and patches thoroughly to minimize the chances of breaking the tree.

2.6 Code Review & Inspection

Code review is the basic mechanism for validating the design and implementation of features and patches. It also helps maintain a level of consistency in design and implementation practices across many developers and amongst the various modules of OpenSAF. OpenSAF has one mandatory and another optional phase of review called: “basic review” and “integration review” respectively. It is the developer's responsibility to coordinate and drive the code review to completion.

The first review phase called the “basic review” is conducted by the module owner or a peer of the module which is the target module for code to be checked-in. One must have an approval from the module owner or designated "peer" of the module where the code is supposed to be checked in. If one’s code affects several modules, then one should have an approval from the owner or designated peer of each affected module. The second review phase called the “integration review” is done by one or more individuals designated by the Technical Leadership Council to verify the fit of the patch or the feature into the broader OpenSAF code base and focus on cross-module issues. It is usually done by individuals who are not necessarily a domain expert of the respective module. One must have an approval from an integration reviewer before the patch can be declared as check-in-ready.

Usually for simpler patches the module owner or peer can recommend that integration review is not required. In such a case a patch can be checked in after the Module owner’s approval.

Once both approvals have been obtained for a feature/patch (with the exception for patches whose integration review phase has been waived) the code is checked in to the tree by one of the committers for the respective module. Responsibilities:

• TLC - Appoint the individuals that do an integration review.
• Committers - Review patches and contributions as described above.
• Developers - Work with the committers to ensure that the contribution is accepted.

2.6.1 Voting Points

1. Maintainers and committers are given wide discretion on whether or not to accept changes from developers. However, when there is serious contention about whether to accept a change a TLC vote can be called on the matter.

2.7 Release Management

Release Management for major releases in OpenSAF is done by the Release Managers chosen by the Technical Leadership Council.

The Release Managers provide guidance to developers as to which changes are important for a given OpenSAF release. They also make a range of tree management decisions. They are particularly active after the trunk is frozen for a major OpenSAF release, and manages that branch until the release is done. During this time the Release Managers watch the check-ins very closely, and generally require that every patch be reviewed by them before it is checked in. Their review is not focused on the technical merit of the patch. The release managers’ review focuses on the importance of that particular fix to the OpenSAF release in question, and the risk to stability and performance presented by that patch. Release Managers often consult with the maintainers & peers before making a call about acceptance of a patch.

2.7.1 Branch Management and Feature Commit Procedures

There is only one release with active development at any given time. This release is managed on a branch called “trunk” which exists forever. Note that in the  OpenSAF Developer’s Guide, the term “master repository” is used instead of a “trunk”, as the former term is commonly used in the Mercurial (a distributed revision control system used in OpenSAF) documentation. Stability on the trunk is of utmost importance, and is maintained through the series of design and code reviews as well as automated build and regression tests as the current release manager sees fit.

Other branches (and sub-branches) come into existence and disappear as various developers see fit. For example, a group of developers working on feature X might have a branch called “FeatureX” and another group of developers working on feature Y might have a branch called “FeatureY”. The management and quality level of those branches is purely up to the developer or maintainer of that branch and is outside the scope of the OpenSAF Foundation. When features have passed all the requirements for commit (design/code reviews, tests developed and passing, etc.) as defined by the release manager, the release manager gives the approval for them to commit their code to the staging branch. Once the there has been a successful build on the staging branch, the code is merged by the release management team to the trunk, and an announcement is sent to the Developer and User communities informing them of the creation of a Beta. At this time, a “tag” is applied to the trunk to identify the Beta. The Developer and User communities will be encouraged to install and test the new Beta, and submit defects and fixes as appropriate. At the end of an appropriate testing period, when the Release Manager feels the Beta has been tested, and has stabilized, a Milestone tag will be applied to the source, and an announcement will be sent to the Developer and User communities. Only 1 large change will be allowed into the trunk at a time (minor fixes from other branches might also go in at the same time) After changes are committed to the branch, maintainers of the other feature branches are expected to merge with the trunk and continue their testing with the merged source code. This ensures that integration bugs (between features) are found early, before they make their way into the trunk. Once the release manager feels that trunk is stable, they can open it up for the next feature commit. The importance that large changes (or new features) should not be committed into an already unstable trunk can not be stressed enough. If trunk is unstable, the only commits should be bug fixes to make it stable again. At the discretion of the Release Manager while approaching a major release the trunk is locked once a day at a certain pre-determined time to make sure that the latest code checked-in builds in at-least a few flavors of the Linux and others Operating Systems. All developers and committers who have checked-in before such a build are “on the hook” and have to be available if anything breaks during the builds. During this time no check-ins except build related check-ins are allowed. After the build process is over the tree is again opened up for further check-in.

2.7.2 Development Model

In reality, Features A and B will be developed concurrently by their respective development teams on their own feature branches. Once a feature meets the appropriate criteria for check-in, the code can be checked into the staging branch as described above. The remaining feature teams are then responsible for merging the trunk into their feature branch and continuing with their development and testing. This process is repeated until all features for a release are checked in. Note that feature branches are temporary, and once the feature is checked into the staging branch (and a subsequent Beta is announced) the feature branch is decommissioned and no longer used. All future fixes should be checked in to the staging branch directly and large future updates to a feature should be done on a new feature branch.

In reality, bug fixes and small updates are frequently going into the trunk, so all feature branches will need to be merging from the trunk on a regular basis to ensure that they never get too far behind in the merge process. It's possible for changes to get lost during complicated merges, so the more frequently merges are done the less complicated they are. In other words, merge early and merge often. However, it is up to the feature developers to decide when the merge will happen for their feature branches based on their internal development guidelines and schedule.

2.7.3 Release Model

Releases will follow a standard model of beta and milestone releases, followed by release candidates, and then a final generally available (GA) release. The release roadmap is essentially feature-oriented, and there will be 1 beta release after each new feature is checked into the trunk. A milestone is announced after a beta is deemed to be stable. Once the trunk is feature complete release candidates will be released, tested, and fixed until the project meets the appropriate release quality criteria. Usually one or more reference platforms are identified on which the given OpenSAF release is verified before declaring that it is “Generally Available or GA”. When the trunk meets the release criteria a patch branch is created off of the trunk and the release is delivered off of the patch branch.

Patch Branches

Patch branches are used as a place to deliver an original release from. These patch branches exist forever (or at least as long as the support life for that release), and updates can be checked into them. Code is never moved from the patch branch to the trunk, and code is never merged from the trunk down to the patch branch.

The term Patch Release is used identify a release from the patch branch. Patch relerases consiste of one or more updates to the barch branch to resolve defects. In general, patch relerases happen under the following conditions:

• there is an important update that needs to be made available (i.e. a core dump in a common operation is fixed, or the split bran scenario is fixed).
• there is a significant number of updates needing to be released.
• there is an update (perhaps low priority) that has been waiting for more than 2 months.

There will be a "release candidate" made available for every patch release. The stable branch will be under change control for the duration of the RC. The RC cycle for any given patch release should not be more than 2 weeks.

All updates made to a stable branch must have the approval of the individual feature maintainer and/or the release manager.

All updates made to a stable branch need to be considered for the development branch. This will make sure the development branch always contains the latest fixes. The maintainer should decide if the patch should be applied to the development branch.

Fixes found and made on the development branch should be back ported to stable branches at the recommendation of the maintainer.

Graphical Release Model Representation

The following picture shows the flow of data going up into the trunk and releases coming out of the trunk:

Note that new branches are not created for the purpose of beta releases. A beta release is essentially a snapshot of the trunk at a given point in time. Bug fixes discovered during beta testing will be checked into the trunk directly, so there is no need for a separate beta branch for each beta release. The trunk should be tagged at the point where we drop a beta release.

Once all features have been submitted, a Feature Complete milestone is created, and the code is tagged. Following the feature complete milestone, Release Candidates (RC’s) are created, culminating in a General Availability (GA) release. The diagram below provides a graphical representation of this process:

2.7.4 Release Version Numbering

The following represents the version numbering scheme is used by OpenSAF

• In general, version number will take the form of x.y.z
  • "x" value indicates overall release version
    • Prior to the General Availability (GA) of any given OpenSAF release, the "X" value is set to 1 minus the expected GA version. For example, if the next expected release is considered to be Release 2 (2.0.0), then the version numbers attached to all betas, milestones and RCs for Release 2 will take the form of 1.y.z.
  • "y" value indicates Release Candidate (RC) version
    • Indicates the RC version
    • For example, a value of "2" indicates RC2 (or Release Candidate 2 for release "x").
    • A value of "0" indicates the release as not yet achieved the "Feature Complete" milestone.
  • "z" value indicates Beta and Milestone version
    • Odd numbers will be used to denote Beta releases
    • Even numbers will be used to denote Milestone releases
    • The "z" value is reset to "0" (and kept at "0" for the remainder of the release) starting with the first RC.

• General Availability releases, increment "x" by 1 and reset "y & z" to 0. All GA releases take the form of "x.0.0".

• Beta, milestone, RC and GA version numbers will be used as tags in the source code tree.

2.7.5 Announcing a new OpenSAF Release

OpenSAF has four types of releases (Beta, Milestone, Release Candidate and General Availability). Information about the completion or availability of these releases will be provided to the OpenSAF community by means of the various OpenSAF e-mail distribution lists, updates to  https://devel.opensaf.org/wiki, and updates to  http://freshmeat.net/projects/opensaf/. The following table lists the person(s) who are responsible for announcing the various types of OpenSAF releases using the above mechanisms:

Release	Announcement Owner
Betas	Developer(s)
Milestones	Release Manager
Release Candidates	Release Manager
General Availability	Release Manager

As stated in the above table, the Release Manager will play a major role in communicating the availability of OpenSAF releases expect for Beta releases. In OpenSAF, Beta releases provide major new features or enhancements updates. As such, the developer is in the best position to describe such updates to the overall community. In addition, these announcements provide well deserved recognition to the developer(s) for their contribution to OpenSAF.

3 Rules Of Engagement

This section describes the process by which Users and Developers can get their functional enhancements and changes included in the OpenSAF Roadmap.

OpenSAF encourages broad and wide community participation. OpenSAF is open to all and provides the same opportunity to all. Everyone participates with the same rules; OpenSAF is also receptive to new ideas for features and enhancements from the community and the membership at large. All OpenSAF project ideas, discussions, minutes, deliberations, plans and roadmaps are open, public and easily accessible.

3.1 Submitting Change Requests

All Users and Developers, irrespective of their membership status in the OpenSAF Foundation, can propose Change Requests (CRs) to be included in the project roadmap. Such CRs should include:

1. Whether the CR is a defect fix or a new feature request
2. The use case for the new or enhanced feature (if applicable)
3. Who will benefit from the proposed feature (if applicable)
4. How does the proposed feature fit the current project roadmap?

All feature enhancement proposals should be initiated through the Change Request repository.

3.2 Feature Enhancement Review Process

All feature enhancement request proposals shall be reviewed by the TLC using the following process:

1. Each incoming feature request shall be assigned a sponsor, who is a member of the TLC. The sponsor is responsible for shepherding the feature enhancement through the TLC processes and procedures, from the proposal stage to implementation.
2. The TLC sponsor shall initiate a on-line review of the feature enhancement proposal. The default duration of review period shall be two weeks. Durations can be adjusted at the discretion of the TLC.
3. All TLC members shall have two weeks to review provided materials, ask questions and provide feedback. The review of the proposal shall be based on the following criteria:
   1. How well the proposed feature fits into the overall project and its roadmap
   2. The significance of the use case and the feature for the user community
   3. Comments and recommendations from the affected module owners
   4. The feasibility to the development and quality assurance plan
4. At the end of the review period, each TLC member company shall vote on the disposition of the proposed feature enhancement. Possible dispositions of a feature enhancement are as follows:

APPROVE	A feature enhancement is approved "as is" and shall be scheduled for implementation on the OpenSAF Roadmap.
REJECT	A feature is rejected.
DISCUSS	Further discussion and review action is necessary. A conference call is scheduled to set up further proposal discussion.
ABSTAIN	The TLC member company abstains from vote.

Final disposition of the proposal shall be based on the number of APPROVE votes. For the feature to be approved, the greatest number of votes received must be to “APPROVE” the proposal. TLC chairs adjudicate the review process and break any ties.

All approved proposals shall be classified as major or minor by the TCCs. All major proposals contingent on receipt of implementation estimates shall be assigned a place on the OpenSAF Roadmap, and will also be subjected to the architecture/design review phase. Minor proposals shall not be tracked on the roadmap, but may still be subjected to the architecture/design review phase.

3.3 Change Request Repository

All change requests are registered in the change request repository. This repository servers as a master "to do" list for technical changes required by OpenSAF. Each approved change request is assigned an implementation owner. The implementation owner is responsible for providing TLC with implementation estimates, and for driving the implementation of the change request to completion, along with the sponsor. If the change request proposal owners included members of the Developer community, those members become the implementation owners of the change request. If the change request proposal owners were entirely from the User community, the change request remains unowned. Developers, who are looking for a project to work on, should consult the change request repository and look for unowned change requests.

3.4 Feature Enhancement Estimates

Once a major feature enhancement proposal has been approved, the proposal originator shall be responsible for providing the TLC with implementation estimate materials. Such materials shall include the following:

1. Description of what changes to the code base the new feature will require
2. Identification of the affected modules
3. Description of quality assurance for the proposed feature
4. Who will do the development and quality assurance for the proposed feature
5. What is the timeline for the development work, and what are the major implementation milestones

Failure to provide appropriate estimates shall result in removal of the project from the roadmap.

4 Appeals and Escalation Procedures

If a developer is not happy with the module maintainer’s decision about a submitted contribution, the developer may appeal to the TLC for re-consideration of the decision. The TLC may stay the decision taken by the maintainer or over-turn the decision as it sees fit.

Occasionally a maintainer may escalate the decision regarding the submission of a patch to the TLC. This happens when the maintainer is not in a position to make a decision about a patch with regards to its overall impact to the project. The maintainer seeks the help of the TLC in such a case.