Archive for the ‘code’ Category

DIASER RoadMap, milestones and release cycle

Tuesday, August 17th, 2010

I though it worth explaining the DIASER release cycle very briefly. Software release cycle is open to interpretation. For example; some web applications are in a perpetual beta state.

I chose to use a fairly formal cycle through various stages, this is appropriate for disaster recovery software:

2005:2006 research/design/testing
2007 – prototype
2008 – alpha-development
2009 – alpha
2010 – beta-1
2010 – beta-2 <– currently
2010 Dec – beta-3
2011 Feb – Release candidate
2011 March – Production (Stable)

2011 April -> new versions

Currently DIASER is beta-2. beta-3 milestone is planned for December 2010. See here for more details http://en.wikipedia.org/wiki/Software_release_life_cycle

The Current ROADMAP – or burn-down in AGILE speak, projects a production release early 2011. http://diaser.svn.sourceforge.net/viewvc/diaser/ROADMAP_DEV

June – November 2010 (beta 3)
==============================
main application
—————-
node migrate adjustments for sudo
network availability checks and reports
option to change node role identity
XMPP(SNMP) interface
prune archives
optional delete volumes from source that have been replicated
sponsored addition to community Linux distributions
listing, upload to: http://perlisalive.com http://pause.perl.org
mixed non sudo and sudo node install, currently either/or
security audit and enhancements: restrict uid command execution
re-factoring – checks against Perl::Critic
more and consistent error handling
tool for filling and auto-replicating retrospective data slots

*diaser_webgui_1
————————
request location and name of conf file
configuration tool
–stats pie chart
–list formatted output
–logs
–retrieve
package as dist, rpm and deb

For later versions
=============
detailed backup software integration study, docs/code?
self healing functionality – if transfers failed or were missed
enhanced archive validation
S3/Google/RackSpace API descriptors for volumes (with price/storage limiter)
automatic node fail-over i.e. a->c if b is down (using node identity
change)

– Damian

Version 1.0.4b2 of DIASER released

Wednesday, July 28th, 2010
Changes: Important tla (AGM) description updated throughout documentation for a more granular meaning, based on changes in IETF-ID LTASP http://bit.ly/aATkDF DIASER is for long term digital archive storage, it securely…

1) Accumulates
2) Geo-Duplicates
3) Manages

  • Engineered storage architecture
  • Exists and operates in dedicated user accounts
  • Flat, human readable storage structure
  • Highly resilient and robust
  • Large volume capacity (TB’s)
  • Low operational and maintenance overheads
  • Manage independently from a Perl enabled workstation
  • Manage long-term archives
  • Migratable nodes
  • Multiple configuration files for multiple installations
  • Perl installer and configurator
  • Powered by rsync and OpenSSH
  • Repair tool
  • Scalable
  • Secure design
  • Simple configuration file and format
  • Standards compliant
  • Stats and analysis tools built-in
  • Straightforward upgrade procedure
  • Use commodity disks for robust storage
  • UTC Time Zone compensation mechanism
  • Works with existing backup infrastructures
  • 3 replicating storage nodes

A new version of I-D, draft-brasher-ltasp-03.txt

Monday, July 26th, 2010
A new version of I-D, draft-brasher-ltasp-03.txt has been successfully
submitted by Damian Brasher and posted to the IETF repository.

Filename:         draft-brasher-ltasp
Revision:         03
Title:                 Long Term Archive Storage Protocol
Creation_date:         2010-07-26
WG ID:                 Independent Submission
Number_of_pages: 13

Abstract:
Long-term archiving storage fundamentally begins with archive data
Accumulation, then Geo-duplication and then Management.  Using
A->G->M, LTASP has been created to solve mid-range and below, long-
term archiving requirements of the small-medium enterprise.  Where
tape has been deployed in the past, LTASP now offers an alternative
solution designed to be more robust and manageable in the long term
than network attached storage devices or simple disk storage alone.

The IETF Secretariat.

This month it’s the Bermuda triangle

Wednesday, July 21st, 2010

I’m not a superstitious person but this observation has got me thinking about triangles again. DIASER, as you may know, is a long-term archive system, disaster recovery, in beta-2 phase. Does this bar chart say anything to you?

bemuda

No, not much. See the whole picture here. From the news you will know that on Tuesday 12th January 2010 a massive earthquake hit Haiti. On 20th April 2010 the BP oil rig explosion occurred. Both locations of these disasters are near, or within, the Bermuda triangle region. The exact location of the Bermuda triangle is open to interpretation. Take a look at the Wikipedia page. Statistics are just statistics and open to misinterpretation; however the graphs above display distinct rise in user activity by March 2010 and now the BP oil disaster has occurred there is a massive drop. Furthermore there was a huge drop in activity in September 2009 shortly after a significant vulnerability in the Linux kernel was discovered on the 13th September 2009. Am I seeing what is not there? Of course, there are many more variables to take into account, so quite possibly. However, it is very possible that the world news affects how different types of software are consumed at any given time. The main point of this is the mind bending, reality law breaking, thought that my life may well have been affected, indirectly, by the Bermuda triangle without even leaving my cosy rural UK based office. The Franco-Bulgarian philosopher; Tzvetan Todorov, states how the breach of our laws of reality might be described in literature, I like it.

– Damian

TransferSummit/UK 2010 – an impression

Friday, June 25th, 2010

Surrounded by stunning buildings lit up by the bright sunlight, Keble College in spotless Oxford was the perfect international setting for TransferSummet/UK 2010. “…a forum for business executives and members of the academic and research community to discuss requirements, challenges, and opportunities in the use, development, licensing, and future of Open Source technology”.

I recently decided to take the opportunity to work for my company, Interlinux Ltd, full time after a three year employment contract within academia. The summit presentations have been of a very high standard and relevant to the Interlinux Ltd business strategy; to primarily provide consultancy and services to UK Higher Education. This summit has been the ideal opportunity to undertake some business development, networking and product awareness; I look forward to seeing the results of my attendance over the coming weeks.

The combination of talking with other attendees and the presentations, has left me with an optimistic picture of the current state of the open source industry and a clearer view of its relationship with academia and open source development communities. I am left picturing a triangle; with academia, the open source business model and open source development communities at each of the corners with a strong relationship along the lines. The Red Hat speaker, Phil Andrews, spoke of the fine balance between generating commercial revenue and investing in the creative and innovative non-commercial communities that are so vital to the generation of open source code. Too much commercial focus then the community can become disenfranchised, too much investment in the community and shareholders become concerned. After talking with academics I can see that academia faces a similar challenge. This summit will assist with the adoption and general acceptance of open source as a viable tool to satisfy many of the immediate and long-term technology requirements within academia and serve as a reference for business. It is apparent, after experiencing the commercial talks and sponsorship, that many of the jagged edges around commercial open source are naturally smoothing out as time progresses.

It is clear to me now, that the open source business, academic and community triangle, mentioned above, has overlapped in history but it is not quite apparent on the surface. Apache httpd server has academic grass roots, commercial application and a strong community with governance provided by the Apache Software Foundation. The president of ASF, Justin Erenkrantz delivered an enthusiastic talk about the foundation. A member of an open source development community could also be delivering commercial open source services and be an academic! Certainly I talked to a number of attendees who resided at two corners of the triangle. The future of continued and more widespread adoption of open source technologies partly depends on finding and building upon common ground between open source service providers and academia. I have experienced plenty of evidence there is common ground the past two days.

– Damian

DIASER 1.0.3b2 has been released

Wednesday, June 2nd, 2010

The changes are as follows:

The manual and other documentation, including images and diagrams, have been updated. The retrieve function now reads zero or more hyphens when used in a volume naming policy.

http://sourceforge.net/projects/diaser/files

Version 1.0.2b2 of DIASER now available

Wednesday, May 26th, 2010

Version 1.0.2b2 Release notes:

26/05/10 – This release features updated documentation; including man page, manual and quick-start guide. SELinux warnings have been removed after further installation and operation tests. A sudo –lock option functionality has now been implemented. Template files for a web GUI tool, which will be released separately, have been added to SVN.

OS files available:

diaser-1.0.2.tar.gz
diaser-1.0.2-1.noarch.rpm
diaser_1.0.2_i386.deb
diaser-dist-1.0.2.tar.gz
diaser-1.0.2-1.src.rpm

http://sourceforge.net/projects/diaser/files/

http://sourceforge.net/news/?group_id=258272&id=286943

Latest flash configure demo for v1.0.2b2:

http://diaser.org.uk/about.html#configure

– Damian

Version 1.0.1b2pre of DIASER now available

Thursday, May 20th, 2010

New features;

stop, pause, resume operations, log parser to condense-display selectable amount of logs from each node. Better stats reporting; disk usage, average diff size, archive storage and list of volumes archived. Time-extend operation adding extra storage structure. Bandwidth calcs can retrieve and build a utility to assisting storage calculations. Makefile, spec and Debian packaging structure; rpm, deb or dist-tarball. Significant documentation updates including a man page and flash configure demo. More bugs have been resolved; time-zone compensation and other minor fixes.

OS files available:
diaser-1.0.1-1.noarch.rpm
diaser_1.0.1_i386.deb
diaser-1.0.1.tar.gz
diaser-1.0.1-1.src.rpm
http://sourceforge.net/projects/diaser/files/
http://sourceforge.net/news/?group_id=258272&id=286943

New flash configure demo:
http://diaser.org.uk/about.html#configure

--Damian


Non-Deterministic anti-reasoning? behind “like never ending tape”

Sunday, April 4th, 2010

To explain the anti-reasoning behind ‘like everlasting tape’ DIASER advertising slogan; this post touches on some mathematics and computer science concepts starting with a small advertising campaign. I needed a way to quickly and cleanly communicate to systems engineers and scientists the purpose of DIASER. I drew a sketch of a strip of tape, that which DIASER is designed to replace, leading into three nodes from a larger data store. The sketch became the image below and the banner logo for www.diaser.org.uk.

right

never ending tape scetch

never ending tape sketch

I was investigating the deterministic and non-deterministic properties of the DIASER algorithm. My systems analyst studies with Red Hat, an architect course, had been teaching me some stochastic, non-deterministic analysis techniques. To help, a truth table exists to improve DIASER analysis on this blog-site called Data Vitality. In computer science and mathematics the Turing Machine describes a model where extremely complex calculations are theoretically possible. Systems can be modelled and can incorporate random elements or processes and are therefore non-deterministic stochastic models otherwise they are deterministic. I am a systems administrator / programmer learning about stochastic analysis, building models which cope with random variations over time. You know how random computer systems can appear sometimes. The weather is complex in Britain due to many random variable values depending on which variables you know of, can or choose to measure and so is the queue at the local super market, seemingly unrelated systems. Perhaps painting a picture is a stochastic activity, an attempt to incorporate and communicate the unknowns of the artist’s vision, an abstraction.

Computer language scientists work with finite instructions sets. The other part of Turing’s model includes an infinite tape consisting of cells to read and write to. Like DIASER can store an infinite amount of data given infinite time. I double checked the Turing machine definition some weeks after I created the DIASER slogan. But given the stochastic nature of my work the, systems administrator / engineering “para-logic” probably works, doesn’t it?

DLB

OSS collaborative development ebook – Chapters 1 and 2 free for review

Saturday, March 27th, 2010

Small scale OSS collaborative development designed to assist, save time and money for and enlighten interested readers. 70 pages for £3.49 (approx. €4.00 EU / $5.50 USD), filled with things I have learned, ideas and tools I have used for DIASER® development over the past few years.

http://walkingwiththeelephants.co.uk/

Free chapters for review http://bit.ly/d4kWxc

Excerpt: “This ebook is designed to help you gain some deeper insight into a small scale, technical Open Source software development collaboration. Perhaps you, the reader, have started, or are part way through, a project; or you are a manager or investor wanting to dig a little deeper into the inner workings of Open Source software development. As the designer, developer and author of a project I recently completed a stage of collaboration that took the software from a prototype to a beta-1 evaluation product. I’ll be talking you through the stages of development from initial conception to beta-1.” – Damian

Walking With the Elephants – Out now.

Estimated cost in £’s of DIASER and DSI coding COCOMO

Saturday, March 20th, 2010

Using the SLOC-by-Language model and an application that does this based on COCOMO model, Person-Months, this is the estimated cost of commissioning DIASER and DSI code development (raw coding hours):

£53,500 DIASER http://www.diaser.org.uk before code tidy and optimisation.

£27,000 DSI http://interlinux.co.uk/wordpress/?p=371

C is a more expensive low-level language, Perl is high level. Of course this does not take in to account the relative success of, or design value to the commercial Open Source communities, Interlinux Ltd and HE partners of the code bases.

The JISC funded component of DIASER, minus administration time, approx 11% of the total DIASER code base at the time of writing. The JISC are among a set of very good development catalysts – see this ebook to find out more.

References:

http://www.dwheeler.com/sloccount [latest version used]
http://www.linuxfoundation.org/publications/estimatinglinux.php
http://en.wikipedia.org/wiki/Source_lines_of_code
http://en.wikipedia.org/wiki/COCOMO

Damian

DSI – invading aliens game

Tuesday, March 16th, 2010

Simple invading aliens game you must stop those aliens from landing on you! Written in C started in 2001. 5 Levels of increasing difficulty, hi-score, sound effects, spooky sound track <space> Fire (unlimited shots available) <right arrow> Move right <left arrow> Move left <escape> Quit the game. I wrote this to improve my C, and to help me with my part time degree studies.

Features
========
Smooth scrolling
High scores
5 Levels
Simple controls
Sound effects
Strategic game play
Linux, Windows and Mac OSX
Customisable graphics out of the box, no re-compile required
Customisable simple definition file for coders
Tiny foot print

Download the Linux rpm / deb, Windows installer or source DSI from SourceForge (Source also available from SF SVN)

View this README for binary installation, run and source code instructions.

See this link CUSTOMISE_DSI to make your own graphics.

If you like this please visit Geo-data replication long-term archive system (WAN vault)

scoresgame2

gamehi-scorelevel1backgroundIntro

The game over 2K lines of C is based on Sam Latinga’s Aliens 1.0.2, part of the SDL library projects released under GPLv2. http://www.libsdl.org/projects/aliens/

Contact me and ABOUT

© Interlinux Ltd. 2005-2010 Registered in England & Wales: Company No. 05588631

IETF-ID LTASP disambiguates DIAP

Tuesday, February 23rd, 2010

The project has moved the IETF-ID from DIAP to LTASP. Long-Term Archive Storage Protocol disambiguates Distributed Internet Archiving Protocol. More accurately describing the aims of the project. The name of DIASER, this GPL software product, will not change. Abstract taken from: http://www.ietf.org/id/draft-brasher-ltasp-02.txt

Long-term archiving storage fundamentally begins with archive data Accumulation, then Replication and then Management. Using A->R->M, LTASP has been created to solve mid-range and below, long-term archiving requirements of the small-medium enterprise. Where tape has been deployed in the past, LTASP now offers an alternative solution designed to be more robust and manageable in the long term than network attached storage devices or simple disk storage alone.

Damian

DIASER Ubuntu LTS sudo workaround – variables defined

Sunday, January 24th, 2010

‘DIR_B’         => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “/home/” },
‘DIR_C’         => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “/home/” },

# additional variables to cope with the Ubuntu (and other distros) forced
# use of sudo; if the USE_SUDO variable is positive then commands that
# require root access will use sudo with the -S switch,
# i.e. echo “root_password” | sudo -S command
# also the username of the account to connect to – instead of root -
# will be used: SUDO_ACCOUNT_NAME_{node}
‘USE_SUDO’      => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “0″ },
‘SUDO_ACCOUNT_NAME_A’ => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “usera” },
‘SUDO_ACCOUNT_NAME_B’ => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “userb” },
‘SUDO_ACCOUNT_NAME_C’ => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “userc” },

# user defined variables for the DIASER filling mechanism / fill_diaser.pl
‘FILL_START_TIME’   => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “10″ },
‘VOLUME_DIR’    => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “/mnt/bkp/” },
‘DIFF_CONST_PREFIX’ => { ARGCOUNT => ARGCOUNT_ONE, DEFAULT => “diff” },

DIAP IETF Internet Draft v11 released

Saturday, November 21st, 2009

DIASER is the GPL software product, there are no plans to make DIAP a software product.

http://www.ietf.org/id/draft-brasher-diap-11.txt

Abstract

DIAP has been created to solve mid-range and below, long term
archiving requirements of the small medium enterprise.  Where tape
has been deployed in the past, DIAP now offers an alternative
solution designed to be more robust and manageable in the long term
than network attached storage devices or simple disk storage alone.
The system provides a well defined structure for storing and managing
long term archives.