Archive for the ‘protocol’ Category

Version 1.0.2b2 of DIASER now available

Wednesday, May 26th, 2010

Version 1.0.2b2 Release notes:

26/05/10 – This release features updated documentation; including man page, manual and quick-start guide. SELinux warnings have been removed after further installation and operation tests. A sudo –lock option functionality has now been implemented. Template files for a web GUI tool, which will be released separately, have been added to SVN.

OS files available:

diaser-1.0.2.tar.gz
diaser-1.0.2-1.noarch.rpm
diaser_1.0.2_i386.deb
diaser-dist-1.0.2.tar.gz
diaser-1.0.2-1.src.rpm

http://sourceforge.net/projects/diaser/files/

http://sourceforge.net/news/?group_id=258272&id=286943

Latest flash configure demo for v1.0.2b2:

http://diaser.org.uk/about.html#configure

– Damian

Version 1.0.1b2pre of DIASER now available

Thursday, May 20th, 2010

New features;

stop, pause, resume operations, log parser to condense-display selectable amount of logs from each node. Better stats reporting; disk usage, average diff size, archive storage and list of volumes archived. Time-extend operation adding extra storage structure. Bandwidth calcs can retrieve and build a utility to assisting storage calculations. Makefile, spec and Debian packaging structure; rpm, deb or dist-tarball. Significant documentation updates including a man page and flash configure demo. More bugs have been resolved; time-zone compensation and other minor fixes.

OS files available:
diaser-1.0.1-1.noarch.rpm
diaser_1.0.1_i386.deb
diaser-1.0.1.tar.gz
diaser-1.0.1-1.src.rpm
http://sourceforge.net/projects/diaser/files/
http://sourceforge.net/news/?group_id=258272&id=286943

New flash configure demo:
http://diaser.org.uk/about.html#configure

--Damian


IETF-ID LTASP disambiguates DIAP

Tuesday, February 23rd, 2010

The project has moved the IETF-ID from DIAP to LTASP. Long-Term Archive Storage Protocol disambiguates Distributed Internet Archiving Protocol. More accurately describing the aims of the project. The name of DIASER, this GPL software product, will not change. Abstract taken from: http://www.ietf.org/id/draft-brasher-ltasp-02.txt

Long-term archiving storage fundamentally begins with archive data Accumulation, then Replication and then Management. Using A->R->M, LTASP has been created to solve mid-range and below, long-term archiving requirements of the small-medium enterprise. Where tape has been deployed in the past, LTASP now offers an alternative solution designed to be more robust and manageable in the long term than network attached storage devices or simple disk storage alone.

Damian

DIAP IETF Internet Draft v11 released

Saturday, November 21st, 2009

DIASER is the GPL software product, there are no plans to make DIAP a software product.

http://www.ietf.org/id/draft-brasher-diap-11.txt

Abstract

DIAP has been created to solve mid-range and below, long term
archiving requirements of the small medium enterprise.  Where tape
has been deployed in the past, DIAP now offers an alternative
solution designed to be more robust and manageable in the long term
than network attached storage devices or simple disk storage alone.
The system provides a well defined structure for storing and managing
long term archives.

OSS Vault building

Wednesday, October 28th, 2009

DIASER – beta-1, we are currently working on project sustainability, making contact with potential trial end users and project maintenance continues…

Vault Building

Vault Building

Read more about DIASER…

VDAP, a vulnerability discovery announce protocol?

Sunday, September 27th, 2009

When you first see this: “Linux Kernel ‘sock_sendpage()’ NULL Pointer Dereference Vulnerability” in an email or twitter it does not mean much at first glance.

When you realise this is a kernel issue if you are a Linux systems administrator you will start scanning, thinking and then you read some more and realise this is serious. Then after 10 minutes the ramifications hit home depending on the systems you maintain and responsibility level on a scale between 1 (a single NetBook running UNR) and 10, plus some off the scale, some may know what I mean.

My sentiments towards the Google security team are ambivalent. They were doing a job and did it well. I feel that the Open Source universe (and propriety) does not have a satisfactory model for announcing and fixing serious security vulnerabilities. The impact on users, businesses, code maintainers, managers all the way through the various components of the producer to consumer can be heavy and I believe the process of announcement to fix could be much much better.

The Google security team announcement (Actually the news was leaked) on the 13th September will have had a negative impact on Open Source communities, like any accident does, but it seems that lessons ought to be learnt.

A simple model could be:

1) Security vulnerability found.
2) Developer(s) contacted privately before announcement is made public.
3) Developer fix privately forwarded to major vendors.
4) Major vendors generate patch and make it available.
5) Public announcement is made.

This point will have been made time and again, could or should a protocol be made law? If a vulnerability has taken a team of top security experts to discover then the likelihood of an individual or organisation finding the same vulnerability in the same amount of time must be slight in general.

I find it difficult to see the benefits of making a vulnerability public before contacting the developer at least. Should a large multi-national like Google be allowed to uncover an error then tell the whole world when it feels like it? Is that ethical? Could it be seen as an act of aggression? (see comment below) (again, as reported above, the news was leaked).

From now on I’m going to make a point of reading more about this process and following moves to change the current vulnerability discovery announce protocol.

Damian Brasher

Original post on Hampshire Linux Users Group

DIASER, Open Source Data Vault Application

Sunday, September 27th, 2009

Open Source Data Vault Application

Proactive archiving… an advanced disk based backup volume management system with triple redundancy for HE and SME. A quick and low-cost way to make an environment more robust and data more accessible by archiving in multiple places. This replication also provides fast retrieval of archived data from all node hosting locations. A Perl installer creates the system. The application provides a decentralised, self-contained and managed storage utility.

Nodes can be dedicated to storage or used for existing services over unused bandwidth. DIASER works in user space over SSH. The software is based on DIAP which is a storage architecture designed to structure months to years of long term sustainable archiving space including retrospective archiving.

The application is beta-1. Please download to evaluate and trial the software, currently being tested with Red Hat Enterprise Linux and Fedora 10. I aim to test with other Linux distributions as well as ActivePerl, freeBSD and MacOSX.

http://www.diaser.org.uk

Damian Brasher

Data Vitality

Sunday, September 27th, 2009

DIAP Rule of Thumb
Circumstantial observation of my email archive, at 272MBytes, having never deleted an email permanently and the file, ../mail, has been in use for 4 years. During this time my available xDSL line Bandwidth has increased, 2004 500MBits/sec to 1GBit/sec, 2008 1GBit/sec to 6GBits/Sec this is about 150% yearly increase whereas my mailbox has increased yearly by about 50%. It is this difference which DIAP attempts to use classing my email record as ‘mission critical’. Other record types will increase at different rates, as will bandwidth depending on location, but probably less than the average yearly bandwidth increase. This idea needs expanding but forms the foundation for the usefulness of DIAP, describing a DIAP rule of thumb. DIAP can also be viewed as a technique.
A pilot survey was run between June 08 and Aug 08 on Hampshire Lug ML named:- ‘[Hampshire] [OT] Identifying the importance and value of data’. This form the basis of a decision matrix tool to help DIAP ® users select data to use in DIAP ® system.Question 2)

“Following on from a survey I posted back in June, see below, designed to help make decisions about data vitality and importance to individuals and organisations, as well as find out a little bit more about the relationship between the importance of data types and their size, I have some results to publish. The survey was exclusive to Hants Lug and deliberately kept low key and has been a very useful exercise even though the number of participants has been relatively small – but very good for such a small readership, quality not quantity.

Thanks to those that took part and those that provided feedback and constructive criticism also thanks to HL ML readers for living with the thread. I will now design a new survey with some sort of incentive and float this to a much larger audience. The results I have are enough to incorporate a DIAP (R) decision matrix on the project website.

6 Participants:

Question 1)

Rate FIVE of these DATA TYPES if they were lost completely how best describes the effect to your users (and or yourself) organisation or home occupants.

Results:

*total damage – cost crippling – traumatic

2 [participants specified] Documents
1 Presentations
1 Photographs
1 Email boxes on a server
2 Code repository
1 Website code

*massive damage – high cost – devastating

1 Spreadsheets
1 .txt files
1 Email boxes on a server

*major damage – very costly – extremely upsetting

1 pdf documents
1 Photographs

*significant damage – significant cost – very upsetting

1 Spreadsheets
2 Accounting Software data
1 Photographs
1 server configuration files
1 music files
1 Virtual machine images

*damage – expensive – annoying

1 Documents
1 pdf documents
1 MS .pst file
1 Email boxes on a server
1 music files
1 ISO images
1 Virtual machine images
1 Code repository

Size the FIVE choices you made in the previous question

Results:

1 MB         100 MB        8 items
>100 MB  500 MB        1 item
>500 MB   1 GB            8 items
>1 GB       10 GB          8 items
>10 GB     50 GB          0 items
>50 GB     100 GB        2 items
>100 GB                        1 item

So from just this small pilot survey I can deduce qualitatively with reasonable certainly that importance of data, subjective to the individual or organisation, does not depend heavily on the data type.

That importance of data in relation to file size is loosely inversely proportional. So the most important files are generally the smallest in size. This is very encouraging information for the DIAP (R) project.

Thanks again to readers and participants.”

Damian Brasher

DIASER: Long term distributed internet archiving application

Saturday, September 26th, 2009

Requested by: http://code.google.com/p/jiscri/wiki/ProjectDocumentation

According to: JISCRI finalProgressPost requirements.

In this post you are required to address each one of the below elements, this post along with your final prototype will be evaluated for its readiness to launch to the end user. If selected, we’ll be using this post as the final “advertisement” that will go in an ‘Argos-like catalogue’ of JISC Software Prototypes (a vision of the future to come!). We hope to put this publication on the desk of as many senior managers in UK HE/FE as possible as well as other potential investors (e.g. NGOs and VCs) in the New Year so please write this Final post with that audience in mind.

Description of Prototype:

Proactive archiving… an advanced disk based backup volume management system with triple redundancy for HE and SME. A quick and low-cost way to make an environment more robust and data more accessible by archiving in multiple places. This replication also provides fast retrieval of archived data from all node hosting locations. A Perl installer creates the system. The application provides a decentralised, self-contained and managed storage utility.

Screenshot or diagram of prototype:

Diagram:

Vault Building

Vault Building

Flash Demo:

http://www.diaser.org.uk/about.html#configure

Screenshot:

DIASER Usage
DIASER Usage
Bandwidth
Bandwidth

Link to working prototype:

http://sourceforge.net/projects/diaser

http://sourceforge.net/projects/diaser/files/

Link to end user documentation:

http://www.diaser.org.uk/files/DiaserDocsv1.1.pdf

Link to code repository or API:

http://diaser.svn.sourceforge.net/viewvc/diaser

Link to technical documentation:

http://www.diaser.org.uk/manual.html

Date prototype was launched: 2nd June 2009

Project Team Names, Emails and Organisations:

PM/Dev: Damian Lajos Brasher

d {.} brasher @ omii {.} ac {.} uk OMII-UK ECS Interlinux Ltd

Neil P Chue Hong: Project advisor

n {.} ChueHong  @  omii {.} ac {.} uk OMII-UK ECS

Dr Simon J Hettrick: Documentation, graphics and marketing

s {.} hettrick @ omii {.} ac {.} uk OMII-UK ECS

Project Websites:

http://www.diaser.org.uk

http://www.diap.org.uk/background_ip

http://www.omii.ac.uk/wiki/DIASER (OMII-UK)

http://walkingwiththeelephants.co.uk (Sustainability)

http://interlinux.co.uk/wordpress (Final Project Post)

PIMS entry:

https://pims.jisc.ac.uk/projects/view/1267

Table of Content for Project Posts

Microblog posts: hash tag #diaser #jiscri

http://twitter.com/#search?q=%23diaser

DIASER SourceForge RSS news feed

News

18/12/09 v0.2.7 beta-1 released

17/12/09 eBook introduction site live

14/12/09 Interlinux Ltd UK Tax year tasks completed

21/11/09 DIAP IETF Internet Draft v11 released

05/11/09 v0.2.6 beta-1 released

30/09/09 v0.2.5 beta-1 released

21/09/09 v0.2.5 beta-1 released

02/08/09 Screencast – demo / mini tutorial released

21/07/09 DIASER beta-1 released

09/07/09 Attended JISC inf11 projects start-up meeting

04/07/09 Community feedback motivate DIASER architectural changes

25/06/09 Manual for DIASER released

04/06/09 Latest 4 month development roadmap, alpha -> beta-2

02/06/09 Build yourself a private storage cloud…

02/06/09 Deployable alpha of DIASER released

22/05/09 Projected release life cycle

22/05/09 Project memory and thinking

21/05/09 Dec 2006 – Mar 2009

17/05/09 Rollout of first fully deployable alpha planned wk 25th May

08/05/09 DIASER moves from alpha-dev to alpha

27/04/09 DIASER funded by UK Joint Information Systems Committee

Other

Download Statistics SourceForge total 1612 since April 2009

      Rank Pages Downloads

Dec 2009 * N/D 8,063 124
Nov 2009 1,585 8,917 183
Oct 2009 1,901 6,200 164
Sep 2009 1,547 11,562 291
Aug 2009 1,317 7,606 423
Jul 2009 1,026 9,588 300
Jun 2009 1,529 4,463 28
May 2009 1,258 4,468 51
Apr 2009 3,429 2,164 53
Dec 2009 * N/D 8,063
Nov 2009 1,585 8,917
Oct 2009 1,901 6,200
Sep 2009 1,547 11,562
Aug 2009 1,317 7,606
Jul 2009 1,026 9,588
Jun 2009 1,529 4,463
May 2009 1,258 4,468
Apr 2009 3,429 2,164

Google analytics for diaser.org.uk :

Analytics_www.diaser.org.uk_20090407-20091222

Analytics_www.diaser.org.uk_20090407-20091222