Archive for the ‘diaser’ Category

DIASER RoadMap, milestones and release cycle

Tuesday, August 17th, 2010

I though it worth explaining the DIASER release cycle very briefly. Software release cycle is open to interpretation. For example; some web applications are in a perpetual beta state.

I chose to use a fairly formal cycle through various stages, this is appropriate for disaster recovery software:

2005:2006 research/design/testing
2007 – prototype
2008 – alpha-development
2009 – alpha
2010 – beta-1
2010 – beta-2 <– currently
2010 Dec – beta-3
2011 Feb – Release candidate
2011 March – Production (Stable)

2011 April -> new versions

Currently DIASER is beta-2. beta-3 milestone is planned for December 2010. See here for more details http://en.wikipedia.org/wiki/Software_release_life_cycle

The Current ROADMAP – or burn-down in AGILE speak, projects a production release early 2011. http://diaser.svn.sourceforge.net/viewvc/diaser/ROADMAP_DEV

June – November 2010 (beta 3)
==============================
main application
—————-
node migrate adjustments for sudo
network availability checks and reports
option to change node role identity
XMPP(SNMP) interface
prune archives
optional delete volumes from source that have been replicated
sponsored addition to community Linux distributions
listing, upload to: http://perlisalive.com http://pause.perl.org
mixed non sudo and sudo node install, currently either/or
security audit and enhancements: restrict uid command execution
re-factoring – checks against Perl::Critic
more and consistent error handling
tool for filling and auto-replicating retrospective data slots

*diaser_webgui_1
————————
request location and name of conf file
configuration tool
–stats pie chart
–list formatted output
–logs
–retrieve
package as dist, rpm and deb

For later versions
=============
detailed backup software integration study, docs/code?
self healing functionality – if transfers failed or were missed
enhanced archive validation
S3/Google/RackSpace API descriptors for volumes (with price/storage limiter)
automatic node fail-over i.e. a->c if b is down (using node identity
change)

– Damian

TransferSummit/UK 2010 – an impression

Friday, June 25th, 2010

Surrounded by stunning buildings lit up by the bright sunlight, Keble College in spotless Oxford was the perfect international setting for TransferSummet/UK 2010. “…a forum for business executives and members of the academic and research community to discuss requirements, challenges, and opportunities in the use, development, licensing, and future of Open Source technology”.

I recently decided to take the opportunity to work for my company, Interlinux Ltd, full time after a three year employment contract within academia. The summit presentations have been of a very high standard and relevant to the Interlinux Ltd business strategy; to primarily provide consultancy and services to UK Higher Education. This summit has been the ideal opportunity to undertake some business development, networking and product awareness; I look forward to seeing the results of my attendance over the coming weeks.

The combination of talking with other attendees and the presentations, has left me with an optimistic picture of the current state of the open source industry and a clearer view of its relationship with academia and open source development communities. I am left picturing a triangle; with academia, the open source business model and open source development communities at each of the corners with a strong relationship along the lines. The Red Hat speaker, Phil Andrews, spoke of the fine balance between generating commercial revenue and investing in the creative and innovative non-commercial communities that are so vital to the generation of open source code. Too much commercial focus then the community can become disenfranchised, too much investment in the community and shareholders become concerned. After talking with academics I can see that academia faces a similar challenge. This summit will assist with the adoption and general acceptance of open source as a viable tool to satisfy many of the immediate and long-term technology requirements within academia and serve as a reference for business. It is apparent, after experiencing the commercial talks and sponsorship, that many of the jagged edges around commercial open source are naturally smoothing out as time progresses.

It is clear to me now, that the open source business, academic and community triangle, mentioned above, has overlapped in history but it is not quite apparent on the surface. Apache httpd server has academic grass roots, commercial application and a strong community with governance provided by the Apache Software Foundation. The president of ASF, Justin Erenkrantz delivered an enthusiastic talk about the foundation. A member of an open source development community could also be delivering commercial open source services and be an academic! Certainly I talked to a number of attendees who resided at two corners of the triangle. The future of continued and more widespread adoption of open source technologies partly depends on finding and building upon common ground between open source service providers and academia. I have experienced plenty of evidence there is common ground the past two days.

– Damian

Version 1.0.2b2 of DIASER now available

Wednesday, May 26th, 2010

Version 1.0.2b2 Release notes:

26/05/10 – This release features updated documentation; including man page, manual and quick-start guide. SELinux warnings have been removed after further installation and operation tests. A sudo –lock option functionality has now been implemented. Template files for a web GUI tool, which will be released separately, have been added to SVN.

OS files available:

diaser-1.0.2.tar.gz
diaser-1.0.2-1.noarch.rpm
diaser_1.0.2_i386.deb
diaser-dist-1.0.2.tar.gz
diaser-1.0.2-1.src.rpm

http://sourceforge.net/projects/diaser/files/

http://sourceforge.net/news/?group_id=258272&id=286943

Latest flash configure demo for v1.0.2b2:

http://diaser.org.uk/about.html#configure

– Damian

Version 1.0.1b2pre of DIASER now available

Thursday, May 20th, 2010

New features;

stop, pause, resume operations, log parser to condense-display selectable amount of logs from each node. Better stats reporting; disk usage, average diff size, archive storage and list of volumes archived. Time-extend operation adding extra storage structure. Bandwidth calcs can retrieve and build a utility to assisting storage calculations. Makefile, spec and Debian packaging structure; rpm, deb or dist-tarball. Significant documentation updates including a man page and flash configure demo. More bugs have been resolved; time-zone compensation and other minor fixes.

OS files available:
diaser-1.0.1-1.noarch.rpm
diaser_1.0.1_i386.deb
diaser-1.0.1.tar.gz
diaser-1.0.1-1.src.rpm
http://sourceforge.net/projects/diaser/files/
http://sourceforge.net/news/?group_id=258272&id=286943

New flash configure demo:
http://diaser.org.uk/about.html#configure

--Damian


Estimated cost in £’s of DIASER and DSI coding COCOMO

Saturday, March 20th, 2010

Using the SLOC-by-Language model and an application that does this based on COCOMO model, Person-Months, this is the estimated cost of commissioning DIASER and DSI code development (raw coding hours):

£53,500 DIASER http://www.diaser.org.uk before code tidy and optimisation.

£27,000 DSI http://interlinux.co.uk/wordpress/?p=371

C is a more expensive low-level language, Perl is high level. Of course this does not take in to account the relative success of, or design value to the commercial Open Source communities, Interlinux Ltd and HE partners of the code bases.

The JISC funded component of DIASER, minus administration time, approx 11% of the total DIASER code base at the time of writing. The JISC are among a set of very good development catalysts – see this ebook to find out more.

References:

http://www.dwheeler.com/sloccount [latest version used]
http://www.linuxfoundation.org/publications/estimatinglinux.php
http://en.wikipedia.org/wiki/Source_lines_of_code
http://en.wikipedia.org/wiki/COCOMO

Damian

IETF-ID LTASP disambiguates DIAP

Tuesday, February 23rd, 2010

The project has moved the IETF-ID from DIAP to LTASP. Long-Term Archive Storage Protocol disambiguates Distributed Internet Archiving Protocol. More accurately describing the aims of the project. The name of DIASER, this GPL software product, will not change. Abstract taken from: http://www.ietf.org/id/draft-brasher-ltasp-02.txt

Long-term archiving storage fundamentally begins with archive data Accumulation, then Replication and then Management. Using A->R->M, LTASP has been created to solve mid-range and below, long-term archiving requirements of the small-medium enterprise. Where tape has been deployed in the past, LTASP now offers an alternative solution designed to be more robust and manageable in the long term than network attached storage devices or simple disk storage alone.

Damian

OSS Vault building

Wednesday, October 28th, 2009

DIASER – beta-1, we are currently working on project sustainability, making contact with potential trial end users and project maintenance continues…

Vault Building

Vault Building

Read more about DIASER…

DIASER, Open Source Data Vault Application

Sunday, September 27th, 2009

Open Source Data Vault Application

Proactive archiving… an advanced disk based backup volume management system with triple redundancy for HE and SME. A quick and low-cost way to make an environment more robust and data more accessible by archiving in multiple places. This replication also provides fast retrieval of archived data from all node hosting locations. A Perl installer creates the system. The application provides a decentralised, self-contained and managed storage utility.

Nodes can be dedicated to storage or used for existing services over unused bandwidth. DIASER works in user space over SSH. The software is based on DIAP which is a storage architecture designed to structure months to years of long term sustainable archiving space including retrospective archiving.

The application is beta-1. Please download to evaluate and trial the software, currently being tested with Red Hat Enterprise Linux and Fedora 10. I aim to test with other Linux distributions as well as ActivePerl, freeBSD and MacOSX.

http://www.diaser.org.uk

Damian Brasher

Data Vitality

Sunday, September 27th, 2009

DIAP Rule of Thumb
Circumstantial observation of my email archive, at 272MBytes, having never deleted an email permanently and the file, ../mail, has been in use for 4 years. During this time my available xDSL line Bandwidth has increased, 2004 500MBits/sec to 1GBit/sec, 2008 1GBit/sec to 6GBits/Sec this is about 150% yearly increase whereas my mailbox has increased yearly by about 50%. It is this difference which DIAP attempts to use classing my email record as ‘mission critical’. Other record types will increase at different rates, as will bandwidth depending on location, but probably less than the average yearly bandwidth increase. This idea needs expanding but forms the foundation for the usefulness of DIAP, describing a DIAP rule of thumb. DIAP can also be viewed as a technique.
A pilot survey was run between June 08 and Aug 08 on Hampshire Lug ML named:- ‘[Hampshire] [OT] Identifying the importance and value of data’. This form the basis of a decision matrix tool to help DIAP ® users select data to use in DIAP ® system.Question 2)

“Following on from a survey I posted back in June, see below, designed to help make decisions about data vitality and importance to individuals and organisations, as well as find out a little bit more about the relationship between the importance of data types and their size, I have some results to publish. The survey was exclusive to Hants Lug and deliberately kept low key and has been a very useful exercise even though the number of participants has been relatively small – but very good for such a small readership, quality not quantity.

Thanks to those that took part and those that provided feedback and constructive criticism also thanks to HL ML readers for living with the thread. I will now design a new survey with some sort of incentive and float this to a much larger audience. The results I have are enough to incorporate a DIAP (R) decision matrix on the project website.

6 Participants:

Question 1)

Rate FIVE of these DATA TYPES if they were lost completely how best describes the effect to your users (and or yourself) organisation or home occupants.

Results:

*total damage – cost crippling – traumatic

2 [participants specified] Documents
1 Presentations
1 Photographs
1 Email boxes on a server
2 Code repository
1 Website code

*massive damage – high cost – devastating

1 Spreadsheets
1 .txt files
1 Email boxes on a server

*major damage – very costly – extremely upsetting

1 pdf documents
1 Photographs

*significant damage – significant cost – very upsetting

1 Spreadsheets
2 Accounting Software data
1 Photographs
1 server configuration files
1 music files
1 Virtual machine images

*damage – expensive – annoying

1 Documents
1 pdf documents
1 MS .pst file
1 Email boxes on a server
1 music files
1 ISO images
1 Virtual machine images
1 Code repository

Size the FIVE choices you made in the previous question

Results:

1 MB         100 MB        8 items
>100 MB  500 MB        1 item
>500 MB   1 GB            8 items
>1 GB       10 GB          8 items
>10 GB     50 GB          0 items
>50 GB     100 GB        2 items
>100 GB                        1 item

So from just this small pilot survey I can deduce qualitatively with reasonable certainly that importance of data, subjective to the individual or organisation, does not depend heavily on the data type.

That importance of data in relation to file size is loosely inversely proportional. So the most important files are generally the smallest in size. This is very encouraging information for the DIAP (R) project.

Thanks again to readers and participants.”

Damian Brasher

DIASER: Long term distributed internet archiving application

Saturday, September 26th, 2009

Requested by: http://code.google.com/p/jiscri/wiki/ProjectDocumentation

According to: JISCRI finalProgressPost requirements.

In this post you are required to address each one of the below elements, this post along with your final prototype will be evaluated for its readiness to launch to the end user. If selected, we’ll be using this post as the final “advertisement” that will go in an ‘Argos-like catalogue’ of JISC Software Prototypes (a vision of the future to come!). We hope to put this publication on the desk of as many senior managers in UK HE/FE as possible as well as other potential investors (e.g. NGOs and VCs) in the New Year so please write this Final post with that audience in mind.

Description of Prototype:

Proactive archiving… an advanced disk based backup volume management system with triple redundancy for HE and SME. A quick and low-cost way to make an environment more robust and data more accessible by archiving in multiple places. This replication also provides fast retrieval of archived data from all node hosting locations. A Perl installer creates the system. The application provides a decentralised, self-contained and managed storage utility.

Screenshot or diagram of prototype:

Diagram:

Vault Building

Vault Building

Flash Demo:

http://www.diaser.org.uk/about.html#configure

Screenshot:

DIASER Usage
DIASER Usage
Bandwidth
Bandwidth

Link to working prototype:

http://sourceforge.net/projects/diaser

http://sourceforge.net/projects/diaser/files/

Link to end user documentation:

http://www.diaser.org.uk/files/DiaserDocsv1.1.pdf

Link to code repository or API:

http://diaser.svn.sourceforge.net/viewvc/diaser

Link to technical documentation:

http://www.diaser.org.uk/manual.html

Date prototype was launched: 2nd June 2009

Project Team Names, Emails and Organisations:

PM/Dev: Damian Lajos Brasher

d {.} brasher @ omii {.} ac {.} uk OMII-UK ECS Interlinux Ltd

Neil P Chue Hong: Project advisor

n {.} ChueHong  @  omii {.} ac {.} uk OMII-UK ECS

Dr Simon J Hettrick: Documentation, graphics and marketing

s {.} hettrick @ omii {.} ac {.} uk OMII-UK ECS

Project Websites:

http://www.diaser.org.uk

http://www.diap.org.uk/background_ip

http://www.omii.ac.uk/wiki/DIASER (OMII-UK)

http://walkingwiththeelephants.co.uk (Sustainability)

http://interlinux.co.uk/wordpress (Final Project Post)

PIMS entry:

https://pims.jisc.ac.uk/projects/view/1267

Table of Content for Project Posts

Microblog posts: hash tag #diaser #jiscri

http://twitter.com/#search?q=%23diaser

DIASER SourceForge RSS news feed

News

18/12/09 v0.2.7 beta-1 released

17/12/09 eBook introduction site live

14/12/09 Interlinux Ltd UK Tax year tasks completed

21/11/09 DIAP IETF Internet Draft v11 released

05/11/09 v0.2.6 beta-1 released

30/09/09 v0.2.5 beta-1 released

21/09/09 v0.2.5 beta-1 released

02/08/09 Screencast – demo / mini tutorial released

21/07/09 DIASER beta-1 released

09/07/09 Attended JISC inf11 projects start-up meeting

04/07/09 Community feedback motivate DIASER architectural changes

25/06/09 Manual for DIASER released

04/06/09 Latest 4 month development roadmap, alpha -> beta-2

02/06/09 Build yourself a private storage cloud…

02/06/09 Deployable alpha of DIASER released

22/05/09 Projected release life cycle

22/05/09 Project memory and thinking

21/05/09 Dec 2006 – Mar 2009

17/05/09 Rollout of first fully deployable alpha planned wk 25th May

08/05/09 DIASER moves from alpha-dev to alpha

27/04/09 DIASER funded by UK Joint Information Systems Committee

Other

Download Statistics SourceForge total 1612 since April 2009

      Rank Pages Downloads

Dec 2009 * N/D 8,063 124
Nov 2009 1,585 8,917 183
Oct 2009 1,901 6,200 164
Sep 2009 1,547 11,562 291
Aug 2009 1,317 7,606 423
Jul 2009 1,026 9,588 300
Jun 2009 1,529 4,463 28
May 2009 1,258 4,468 51
Apr 2009 3,429 2,164 53
Dec 2009 * N/D 8,063
Nov 2009 1,585 8,917
Oct 2009 1,901 6,200
Sep 2009 1,547 11,562
Aug 2009 1,317 7,606
Jul 2009 1,026 9,588
Jun 2009 1,529 4,463
May 2009 1,258 4,468
Apr 2009 3,429 2,164

Google analytics for diaser.org.uk :

Analytics_www.diaser.org.uk_20090407-20091222

Analytics_www.diaser.org.uk_20090407-20091222