RoadMaps/2010-Q1Q2-RoadMap

Connexions Technical Roadmap

January - June 2010

(updated Jan/12/2010)

Technology plan summary

Enterprise Rhaptos: For the first half of 2010 the Connexions technical team will be delivering Enterprise Rhaptos, supporting the first adopters, and ironing out any technical glitches that the consortium and adopter community identify.  While supporting the early adopters, the Connexions team will be refining the new Enterprise Rhaptos development processes and making sure that the documentation works for the early adopters, updating and organizing as necessary. The Enterprise Rhaptos Developer Kit and Enterprise Rhaptos Independent Repository will be delivered prior to the Connexions Consortium meeting and Connexions Conference 2010. Enterprise Rhaptos Virtualized (a version of the software runnable on Amazon Web Services) will be delivered prior to the Open Courseware Consortium, May 5-7 meeting in Vietnam.

Performance: Over the next six months, Connexions will be working to improve two aspects of performance: the time to deliver published content over the web (module and collection viewing times) and the overall times that authors wait for pages to load and actions to occur while they are working with their accounts and content. Content viewing times will be addressed through caching improvements, and the authoring times will be addressed by investigating and eliminating bottlenecks. We will use a combination of external contract development and internal advising and quality assurance/testing in order to make significant (reduction by 1/2) and lasting gains in these two areas. The performance work is being funded by Connexions and funding in the Community College Open Textbook Collaborative grant to support tools and software enhancements that will benefit community college faculty and students.

Big Digital Machine and Community College Open Textbook Collaborative: The Connexions technical team will be engaged in software development for two separately funded projects: the Big Digital Machine (BDM) through Indiana University, and the Community College Open Textbook Collaborative (CCOTC). For the BDM, Connexions is enhancing the Google Analytics statistics for repository usage, building an importer from PKP's open journal system into Connexions modules to support a pipeline from research to textbook education, and writing a specification on Connexions content preservation and backup using Duracloud services. For CCOTC, the team will be addressing authoring performance and integrating the Siyavula developed Express Edit software. A new version of the Collaborative Statistics textbook will be created using the custom PDF pipeline that was developed to properly format homework and labs. In the second half of the year, CCOTC Connexions developer funds will be used to integrate the custom PDF features into the automated PDF production.

Siyavula partnership support: The Connexions team will be providing development support to the Siyavula project to complete several software projects already underway.  The Connexions team will be providing quality assurance and testing, code completion support, and release management for two existing development projects:  1. a bug-fix release to the lens and tag enhancements that are available for Siyavula accounts, 2. Express Edit, a quick way to check out a module to edit or derive a new copy while viewing the published module. Connexions will also be conducting an evaluation of the effort needed to bring open lenses, vocabulary-based tags, and discussion forums to the entire Connexions authoring community.

CNX.org maintenance: On an ongoing and continual basis, Connexions maintains the cnx.org hardware, software, backup and logging systems, health and performance monitoring, and development environment (software versioning, bug tracking, system documentation, automated build and test system, account maintenance), and the web presence for Connexions social media and the Connexions consortium. Connexions also performs all code release management for Rhaptos as used on cnx.org and provides technical support to users and investigates and repairs critical bugs.

Consortium Technology Committee : Future development mechanisms: Connexions and the Technology Committee of the Consortium will be exploring new consortium-based ways to carry out development. Siyavula will be proposing a model for joint contracts between funders, contract developers, and the entities responsible for Connexions repository integrity that specifically include funding all aspects of the software development from contract to live release. They will also be proposing a novel mechanism for funding designs and development effort estimates.  

Knowledge Transfer : The Connexions team will be continuing to build upon the documentation and bug tracking system, Rhaptos Trac, introduced in 2007. The team will concentrate on making sure that knowledge is easily digestible for external development. The knowledge transfer will include the build and install using Enterprise Rhaptos, the source code and how each product works, the testing system and the standards for quality assurance, and this roadmap document.  Internally, we will be continuing to document the System Administration and Production tasks performed by Rice University's Connexions Systems Engineer. Specific administration documentatoin will be conducted in the private area of rhaptos.org, which will used internally only.  Public information currently on rhaptos.org will be transitioned to Rhaptos Trac.

Contingencies in the technical road map

The software development time line shown below includes some slack for two things that we can't completely predict: the extent of Enterprise Rhaptos support requirements and the number of personnel. Several of the software projects are subject to contingency risks outside our control. Detailed contingency information is provided at the end of the document regarding the Express Edit feature, the peformance improvement work and the open lenses and vocabularies. 

Time Table for 2010 Development

Project Name
Design
Develop
Test
Launch
Siyavulala Phase 3
Done
Done
(External)
Done
Jan/Done
Enterprise Rhaptos (ER) Developer Kit
Done
Done
(External)
Jan
Feb
ER Test System Infrastructure Done
Done
(External)
Jan
Feb
ER Virtualized
Jan
Feb-Mar (External)
Mar-
Apr
Apr
ER Test System Test Suite
Done
Feb - Jun
Ongoing
Jun
Siyavula Express Edit and Reuse
Done
Done
Feb
Mar
Consortium Development Support
Review development proposals
Support external installs and developers
Enhance documentation
Jan-Feb
Mar-Jun
N/A
N/A
BDM Google Analytics for Download
Mar
Mar
Mar
Mar
BDM OJS to Connexions Importer
Feb
Mar-Apr
Apr
May
BDM Write Duracloud Connexions Services Spec
N/A
Mar-Apr
N/A
May
CCOTC New PDF of Collaborative Stats
N/A
Apr
N/A
May
CCOTC Authoring performance / Catalog Removal / Separate hardware
Jan-Feb
Feb-Apr
May-Jun
Jun
CCOTC Performance Content Caching
Jan-Feb
Feb-Apr
May-Jun
Jun
Knowledge Transfer
    System Administration
    All development to Trac/Dev Blog/Consortium  
Feb
Feb
Feb-Jun
Mar
Siyavula Open Lenses and Vocabularies
    Usability Design and Evaluation
Evaluate
Design UI - Mar
Depends on evaluation and funding
Depends on staff/funding
Unknown
CCOTC Integrate PDF customizations
Jun
Jul-Aug Sep
Oct
CNX.org maintenance
Ongoing
Ongoing
Ongoing
Ongoing

Development Project Details

Siyavula Phase 3: Small Bug Fixes (Trac Milestone)

Upfront, financed by Shuttleworth's Siyavula project, fixed 29 bugs in their original code release for open lenses (anyone can add content) with vocabulary-based tagging. The code fixed bugs important to siyavula authors, as well as bugs that Connexions Q/A found that were simple to address. This round did not address usability bugs. Connexions development manager has been testing their fixes as they come in and in January the Connexions team is doing code merging, final QA and repair, and rolling this code out. Because it was developed prior to the new Enterprise Rhaptos system, it must rollout before Enterprise Rhaptos.

Enterprise Rhaptos (ER) Developer Kit: Build System and Test System Infrastructure  (Trac Milestone Documentation)

Enterprise Rhaptos streamlines installing, deploying, and developing Rhaptos' and Connexions' open-source software and enables development teams from the Connexions consortium to enhance and improve the software. 


Contractor, Enfold, has delivered the code and the Connexions team will be merging it in January in preparation for release prior to the consortium meeting. The entire team is participating.

Enterprise Rhaptos Virtualization (Trac Milestone)

Virtualization on the Amazon Web Service (AWS) of the Connexions platform will allow production environments to be scaled as demand increases and decreases, will be fault tolerant regarding hardware malfunction, will support flexible testing, and will make it easier to provide demonstration instances of the software when needed. Virtualization on the Amazon cloud will also make providing hosting services for Rhaptos instances more flexible.

The contractors are starting on virtualization at the end of January after the release of Enterprise Rhaptos. Testing of the delivered code will occur at intermediate dates yet-to-be specified in the Statement of Work and final test and deployment will occur in April.

Enterprise Rhaptos Test Suite (Test Suite List)

Enfold delivered a test system infrastructure with basic tests. In order to do consortium development, the team has identified a suite of regression tests that will be needed to see if new features break the existing system.  We started implementing these in December and will continue to implement them through the spring.

CCOTC and Siyavula Express Edit (Trac Milestone)

Connexions designed a new feature to make it easy for authors to check out or derive a copy of a module from the published view of the module. Currently, new Connexions visitors are completely unable to figure out how to derive copies. Siyavula chose to fund Upfront to develop the designed feature and Upfront finished it late in 2009. Express Edit will be adapted (merged) to the Enterprise Rhaptos system and then Connexions will Q/A, repair, and release this new feature unless some major problem is uncovered in the delivered code.

Consortium Development Support

Review development proposals : Connexions has written an initial draft of a development proposal format and Upfront is using it to do functional specifications and cost estimates for designs that Siyuvula thinks are important for Connexions and Rhaptos. These and hopefully other proposals will come in and Connexions will help review them with the consortium. Development of these sorts of procedures will be one of the critical tasks for the Consortium Technology Committee.


Support external installs and developers : We already know that several teams are either already starting to install or rapidly planning to install the new Enterprise Rhaptos. We have reports from two external teams that were able to successfully install the system. We sought out one of the teams, and the other tested the system independently and reported the results to us. The team will support this process as it begins and fix any issues that arise and make the documentation easier to use as we get feedback.

Enhance documentation (trac, devblog, rhaptos.org) With more external users, we are migrating all documentation to trac, creating a developer blog for ongoing dialog, and transforming rhaptos.org to be an internal only asset for sensitive system admininstration documention.

BDM Google Analytics for Download (Trac Milestone)

Indiana University is spearheading a project called "Big Digital Machine". An element of the BDM is to link scholarly works from reasearch to publication to the classroom, with Indiana, Connexions, Duraspace, and Public Knowledge Project as initial participants in the latest phase of the BDM. So that the impact of content can be meaningfully presented and compared no matter what part of the BDM the content lives in, each project will implement uniform and comparable statistics.  Connexions already uses Google Analytics to track page views, and the group felt that tracking downloads was a critical addition since downloads represent important use cases and indicate real interest in the content and potential reuse of the content. Connexions will extend GA tracking to include tracking PDF downloads.

BDM OJS to Connexions Importer (Trac Milestone)

Connexions and PKP will work together to create a mechanism for journal publishers or journal article authors to publish articles to Connexions. This bridge will facilitate the reuse of peer-reviewed journal-published materials in educational settings, courses and textbooks, and will increase the ways that content can be moved and converted between different parts of the Big Digital Machine.  PKP has a Word/Open Office feature for some journals and Connexions has a Word/Open Office importer to convert to Connexions' XML formal. The bridge will build on this natural pathway.

BDM Duracloud Connexions Services Spec (Trac milestone)

Content replication and preservation of repositories will be an essential component of the Big Digital Machine system and Duracloud targets these services. 

Connexions will work with Duracloud personnel to determine how Duracloud services can be used for content preservation. Connexions will write a specification that details the workflows that can be supported and how the services can be used. Software development needed to present Connexions data to Duracloud and provide API's and workflow for content preservation will be designed, specified and estimated for cost and time.

CCOTC and Connexions Performance Improvement (Author performance milestone) (Cache milestone)

Options: Catalog removal / Separate authoring hardware / Content caching / Feed usage optimizations / Load balancing

Kathi has been talking to the team as well as contractors about potential significant performance enhancements that would require non-trivial development efforts. These are efforts that we have not been able to conduct in-house while the code-base was rapidly changing, and especially while we were having external developers refactor the code for Enterprise Rhaptos. We are planning to use a combination of external development and internal consulting and quality assurance during the next 6 months to improve both authoring and reader performance.


For authoring, the Plone catalog is a suspected bottleneck and Shuttleworth/Siyavula is funding an experiment in January to determine whether optimizing or replacing the catalog would result in a significant improvement to authoring performance. They will provide the report will to the consortium at the annual meeting.


The performance work is being funded by Connexions and funding in the Community College Open Textbook Collaborative grant to support tools and software enhancements that will benefit community college faculty and students.

CCOTC New PDF of Collaborative Stats

The Collaborative Statistics textbook uses a speciailzed version of the print system in order to handle the printing of homeworks and labs differently from our standard print pipeline. A new version of the PDF will be created to incorporate the latest updates.

The CCOTC budget includes money for integrating this custom PDF generation functionality into the standard print pipeline later in the year.

Knowledge Transfer

The team has been working on documenting knowledge on the trac system since it was installed in summer of 2007. The Connexions team will be concentrating on making sure that knowledge continues to be encoded for external development. Ed instituted formal knowledge transfer sessions with the team and has been transcribing those sessions onto trac. The team documentation falls into the following categories:
  • Build and install using Enterprise Rhaptos (Enfold initial, Connexions ongoing)
  • The source code and how each product works (Ed)
  • The testing system and the standards for quality assurance
  • The user interface designs that have been done but are not scheduled for development and implementation. (Author profiles, Search and browse lenses, Module authoring workflow, Collections of unpublished modules including full preview system, Print parameters for collections the modules within them, Collection editor with drag and drop (designed by Upfront with Manpreet's help))

Internally, we will be documenting
  • the System Administration tasks performed by Ross Reedstrom, even more than before
  • administration of trac, the upcoming developer blog, and the consortium website

Siyavula Open Lenses and Vocabularies Wide-Release Evaluation (Milestone)

We have identified several user interface issues that might need to be resolved before releasing open lenses and vocabularies as-is to the community-at-large. We were hoping to have an independent evaluation of the functionality by the Wikiwijs project who were considering getting into the Siyavula beta themselves. If a big potential partner is satisfied with the functionality as-is that would indicate that others would be also. This evaluation may yet occur and would be quite valuable. The individual concerns are listed in milestone discussions.

CNX.org maintenance

Connexions maintains the cnx.org hardware, software, backup and logging systems, health and performance monitoring, development environment (software versioning, bug tracking, system documentation, automated build and test system, account maintenance), and the web presence for Connexions social media and the Connexions consortium. Connexions also performs all code release management for Rhaptos as used on cnx.org and provides technical support to users and fixing critical bugs.

The Systems Engineer is responsible for ensuring that hardware and software is configured and operating correctly and that the hardware and software is replaced/repaired upon failure and upgraded with current processor capabilities. When technical issues are reported about the cnx.org site, the SE investigates the problem, provides support back to the user to resolve or work around the issue, investigates and fixes critical bugs.

The SE also maintains the onsite and offsite backup of the repository and maintains two systems for monitoring health and performance, one for alerts and alarms and another for long term tracking and problem diagnosis. The Connexions development environment consists of LDAP account and permissions, development hardware and software, an SVN repository of the software, the trac bug tracking system, the rhaptos internal system administration system, and the automated build and test system for Enterprise Rhaptos. The SE also maintains and updates the Connexions social media presence and the consortium website and accounts.

CCOTC Integrate customizations into the PDF system (Milestone)

Adapt the print and web display tools to meet identified needs of community college faculty
and students for using open textbooks in Connexions. The following adaptations were identified in the pilot project that converted Collaborative Statistics:
  • Closer matching of the online and print numbering systems.

  • Support for homework and laboratory modules so that

    • Pagination starts on a new page.

    • Numbering starts over from 1.

  • Support for appendices numbering appropriately in the table of contents in print.

Contingencies in the technical road map

The software development time line shown above leaves some slack for two things that we can't completely predict: the extent of Enterprise Rhaptos support requirements and the number of personnel through June. The roadmap is also subject to specific development contingencies that we cannot entirely eliminate discussed more fully under Express Edit, Performance, and, Open lenses and vocabulary based tagging wide release. Details are below.

Enterprise Rhaptos Adoption Extent and Initial Support Requirements

It is critical to provide initial support to the fledgling consortium and to iron out any documentation and procedures that cause problems for external developers while we still have a team in place with long-standing Connexions experience. While the consortium, outside companies, and Connexions can provide paid support after June, it is important to have these months with the Connexions staff to make sure everything works smoothly.

Personnel loss

The second consideration is that we are likely to lose personnel that have found other positions before the end of June transition to the consortium.

Express Edit

The Connexions team will adapt (merge) the Express Edit code developed by Siyavula to the new Enterprise Rhaptos system and then Connexions will Q/A and attempt to release this code. Contingencies include whether major bugs are found during Q/A and whether Siyavula/Connexions/or the Consortium have the funds and time to repair those bugs.

Performance

Our goal is to reduce average content viewing times by half or more and to reduce authoring wait time per action by half also. While we feel there is a very realistic chance that we can achieve these gains, it is not possible to guarantee them. Caching is certain to achieve such gains and likely even more, but the project might prove too complex for the given time frame, proposals might exceed our budget, or contractors could fail to deliver.


Because the authoring performance slowed down dramatically with the size of the published repository, unloaded development instances slowed down also, and yet authoring in the new empty Enterprise Rhaptos system is quite zippy, we suspect that the Plone catalog is to blame and that it can be optimized or even replaced. Furthermore, if the catalog turns out not to be the problem, we suspect the problem is deducible and reparable because we now have a clear evidence and differentiation points, and we have combined expertise of several teams available. Nothing is guaranteed, however; the catalog might not be the problem, no large bottleneck might be found, and/or the problem could prove too expensive to tackle.

Open lenses and vocabulary based tagging wide release

Our current 6 month development plan does not include releasing open-lenses and vocabularies to the full Connexions community, becaues while the new functionality is quite a boone to the Connexions system and will be very useful, our UI team identified several potential user inteface problems, If they must be fixed before a wide-release, the design and development required could require a 3 person-month effort. The Siyavula project does not need the enhancements because they have completed the creation of their lenses and vocabularies and no longer need to work with the UI for those features. Siyavula has generously offered to let others into their beta program, however, so they can see exactly how it works.


Since we haven't had any beta testers yet of the functionality we don't know if these concerns must be addressed before wide-release. If they must be fixed before release, funding and expertise would be needed for design of any repairs needed, user interface mockups (CNX), development, Q/A and release to cnx.org. We don't know if that funding could be obtained.


If we do not have an evaluation that indicates that the code should be immediately released and we do not have funding to repair it, then the Consortium will decide how to proceed.