Blog‎ > ‎

RaspPhotoPipe - En

++ = RaspPhotoPipe

A few years ago I buy a Photoduíno that I gave little use, mainly because inside home I do not have many opportunities to take some studio pictures. Sometimes when we take photos outside, would be nice to have a bigger screen to seed the photos than the photo machine LCD. This Christmas, Santa Claus got me a tablet, but because is not possible  connect the tablet directly to the camera (as far as I know), I decided to join a Raspberry Pi through the middle to give more powers to the camera.

So I decided to start a new project, and after some searching, I found someone with the same idea, but does not say much how to do it. I'm a fan of the open source
philosophy, so I will share the steps of my project. From electronics I don't know anything, but I'd like to learn something. First  I'll assemble the easy parts, like configure software and join cable, the try to integrate some sensors to the Raspberry Pi, as I did with Photoduino.

My project goals are:

1st. Find a Raspberry Pi;
2nd. Connecting the Raspberry Pi  with photo
machine and take pictures via gphoto;
3rd. Join the tablet, first make the two communicate, preferably not being dependent on an AP;
4th. Arrange a form in the tablet to see photos just taken with the machine;
5th. Expand storage space ...

So far everything (hopefully) will be easy .... just join software and cables, then begins the part of involve some electronics:

6th. Add batteries to power out of the gadget;
7th. Put everything together to be transportable.
8th. Start adding sensors ...  :)

Besides going to blog all steps of my adventure, I will also add information on how much I will spend on the project.



Storage Expansion - Part IV

posted 4 Aug 2013, 08:09 by Flip Pipe

In order to RaspPhotoPipe be useful, at least on raw file should be copy below 1 second. With my last results, I could copy a raw file (about 20Mb in my camera) in about 2 seconds. It wasn't very bad, must should be better, so I start to find out how to get better from my raspberry pi.

Something I learn from this, was: bigger not always are better, because this is a small device and it is very easy to reach resource starvation.

First, get the best from USB disks, with the lower resources.

According with this FAQ there is a configurable value in kernel to get more throughput from USB. It is the max_sectors, which limits the number of bytes in each command.

USB

In my configuration, max_sectors will be 192, because, even asking more bytes, I will not get much more speed.


Next step, how kernel handle the I/O calls.

Linux kernel implements 4 different algorithms to read/write data to the device, and those are know as schedulers. You can find here in StackOverFlow a brief explanation.


But in Raspbian, just 3 are in the kernel: noop, deadline and cfq.

Scheduler

Don't be mistaken by the chart, because if I didn't change the scale, almost there is no difference between them, so I don't think change this will have a big difference in the end.


Next step, how to improve the RAID, and I start by the building blocks of RAID. What is the size of the smallest amount of data to be written in the disk (chunk size) and the RAM used to cache the data (stripe_cache_size)


After running a scripts varying this values, I find out the best values are:


Chunk size of 32 bytes and a stripe cache size of 4096 bytes. A chunk size of 8bytes could have better writing throughput, but the reading speed if must worst.


Total RAM allocated to writing cache it will be 4 disks x 4k x 4096 bytes = 64MB, according with this comment.

Write




The last step, is the file system.



So far, all my tests where doing DD from /dev/zero to the disk. But with this tests, I've two problems: first it will not be the way I will copy the data to disk in real usage and second, this is highly resource intensive to the raspberry pi. If the processor it is occupied to generate the zeros, it can not using that time to process the data in the raid.


So I change the way to do the tests. So I my next step was copy the files from the ram of raspberry pi to the RAID. And I also stop doing testes with stripe cache lower than 1024.


A big change in the numbers, right? But this numbers are some how fake, because the origin of the data is in RAM, and the destination, because the cache of Raid, it is also RAM. So from this graph, I only can conclude xfs will be my first choice, because it is more consistence in the tests even with worst performance than btrfs.


By the way! One hypothesis I've done in my last post, is if the mdadm and FS modules are compiled in kernel it will be faster... well there you have the answer... No visible changes, so, don't wast time recompile the kernel just for this.



So, my last try to have some real world results, was to copy files from a Compact Flash to the Raid. Almost no difference between the FS.


So many hours running tests, to reach a conclusion: Probably my bottle neck it is in USB it shelf rather than in the RAID performance.



Storage Expansion - Part III

posted 4 Apr 2013, 05:31 by Flip Pipe   [ updated 4 Apr 2013, 14:54 ]

In last post, I've tell I will work with 3 USB flash drive in RAID 5 configuration. It will give something around 30 MB/s of writing throughput.

Meanwhile, the components to Raspberry Pi has arrive a I've been able to start doing some tests in it.

Disappointment... the first test with 3 pens in RAID 5 just gave me about 5MB/s... where are my 30MB/s???

Oh... I forgot do the fine tuning... lets increase the stripe_cache to about 8M and redo tests...

Luck... no, no luck at all... it goes down to less than 3MB/s.

Now I've realize I will need to do a bunch of tests again to see what are optimal values of the stripe_cache and what else can be fine tuned in order this works... at least I want 20MB/s.

Googling around, I've find, the value of chunk size when the raid is created also makes effected in the global performance.

So, I've done some tests, varying the chunk size and the stripe_cache. And this are the results:


As you can see, the fine tuning is really important to writing... with the same setup, you can go from ~3MB/s to ~12MB/s, 4x more.

The readings, well, the stripe_cache doesn't affect the performance, just the value of chunk size, so this graphic just show the evolution of the throughput changing the chunk size.


How can I improve the performance of writing... adding more disks? The hub I've used so far, didn't have external energy and when I connect the 4th pen it stop work. So I've to buy a powered hub... and I've found one with 7 usb ports :)

New hub, it will make difference in the previous results?


No, it doesn't make difference... and adding a 4th pen increase the performance. Good... should I be able to increase it to 20MB/s?

You are right... I've done more testing, with the same previous philosophy and this the results:

 
No, I can't reach 20MB/s... Probably with a powered hub with 7 ports and with 6 pens in hand I could reach the 20MB/s but will be one more Amp to provide to the system working with batteries, so, in the future i could do some tests with 6 pens, but for this project I will used just 4 and work with it.

All this tests have been done without a file system in the RAID, so, my last test was to see how much the performance will decrease with a file system... and this are the results:


More degradation... less 4MB/s... this is not good... more improvements must be done. The RAID software it is not static in kernel but in modules... and lots of it. So will try to improve it compiling the RAID directly in kernel.

For now, I will stop trying to improve this... At this point I've all I need to do the first stage of the project. I've gphoto comanding my camera. I've WebCamCon working. I've a wireless usb dongle doing as AP with my table connected to it. I've the (not good) expanded storage... Its time of put all thinks together.

Storage Expansion - Part II

posted 23 Mar 2013, 12:53 by Flip Pipe   [ updated 23 Mar 2013, 13:40 ]

As I wrote in February, one of the purposes is to expand the CF card of the camera. After some weeks of experiences finally I choose something for my project.
  • Three USB 3.0 Flash Drive in RAID 5.

This decision was made after about six weeks of investigations to reach this statement before.

What type of USB pen drive to use

My first and probably most obvious decision was to use USB 2.0 pen drives, and there are two kinds of this: UDP and COB (Chip on Board).


A description about the differences can be found here, but it don't say about life time and throughput. I try some help from StackExchange but without luck so far.

At this point the only comparison I've was two usb pens. Using Disk Utility from Ubuntu I got some results:

 COB UDP
 
COB
 
UDP

With this results and because they are smaller, I focus in UDP.

Where to Buy

My first thought was buy it directly from China and from the manufactures to be cheaper and to buy it with any casing.

I find a site of business to business: http://www.alibaba.com/

I've contact about 6, all answer to my questions, but in the end, the MOQ (minimum order quantity) was 50 unities, and at the best I will use 6 in a more elaborated RAID configuration.

One of the manufactures, point me the wholesale version of Alibaba: http://www.aliexpress.com

But, without casing I didn't find any product, to be cheap, the MOQ was 10 unities.

So lets check the more conventional stores.

After some hours in MyMemory comparing flash drives, I choose a Kingston DataTraveler USB 3, 32 GB.

"The price is identical to USB 2, even being a lousy USB 3, should at least have the same performance as USB 2." This was my thoughts to buy it... and buy 6 of them.

By curiosity, MyMemory send me each of them in a different package. Not very ambient friendly.

When it arrives, I've done some tests with Disk Utility, and for me was a little disappointing.

Kingstone

As you can see, reading has a good throughput, but writing it is as good as my UDP USB 2.0 flash drive.

Now, with 6 flash drives, I will not buy others. But there is a peak of hope... like the first peak in writing. Probably, that peak can be enough for in the overall RAID give some gain.

What RAID level to choose

I will not explain the differences between RAID level because they are well documented in the web.

RAID 0 - Gives the best write performance and best price per GB, but it is not usable because if one pen fail, all RAID will fail, but will test it for comparison.

RAID 5 - Good price per GB, but it lacks write performance, but how much?

RAID 50 and 05 - At a little higher cost of price, but joining the striping with mirroring, probably will get bester performance than RAID 5.

RAID 10 and 01 - The most expensive in consideration of price, but it will have a great performance... well lets test it.

To do the testing, I've connect the 6 pens to the computer and slice each in 6 partitions



With it, I've created 6 RAID drives, each one with 6 partitions, one of each pen and in the end have this scenario:







The first 3 are RAID levels directly supported by mdadm, but the other three are combinations with the RAID levels because they are not directly supported by mdadm.

With this configurations I get this total size of the volumes


And at this point, the winner is RAID 5, followed by their brothers 50 and 05.

Next point was test the writing throughput. To do that, I've done two tests... one with dd coping 4G from /dev/zero, and other more realistic for their use, created 7 directories with 3 sets of pictures, one in raw and other in jpeg. Then with rsync, copied it from memory and between copy of each directory I wait 5 seconds. The last test is closest I remember to simulate the real use. All the tests were done three times to check the consistency.

And the firsts values with dd were disappointing.


All RAIDs with stripping (level 5) the results where miserable. During the tests, I've collected some data how was the writing in each partition with dstat. You have that data and some more graphics in the files of this post. That information allow the creation of this graphic


But how it will compare with a test more similar with the reality.


Any conclusions with this tests... I notice while taking the note of this values, they are very similar during the sets and the rounds of the tests. As you can see, there are many cell with values ~107MB/s, ~35MB/s and ~15MB/s in all RAID levels.


With this results, RAID 5 have more chances of been choose, but why this results. Probably some kind of RAID memory cache before the data be sent to the physical drive.

I've tried to search this question googling a bit, but I find something more useful.
There is a parameter in RAID 5 which affects the performance: stripe_cache_size

In Stack Exchange there are one answer

http://askubuntu.com/questions/19325/improving-mdadm-raid-6-write-speed

In this blog it has a graphic how the value of this parameter affects the write performance

http://peterkieser.com/2009/11/29/raid-mdraid-stripe_cache_size-vs-write-transfer/

The why... I didn't find...

From the default value of 256 I've change to 8192 the redo the tests with RAID 5... check the differences



Write speed during time...


WOW... great difference... with RAID 5 tuned, there is no more questions which RAID level I will use in my RaspPhotoPipe.

But all this tests were done in my desktop, not in the Raspberry PI. So the next post storage will be configuring and fine tuning RAID 5 in RaspPhotoPipe.

Portable power to my Raspberry Pi

posted 22 Feb 2013, 15:11 by Flip Pipe

Today I found another piece to my project... which is add an external power source to use it outside. It is better than add a power source, is to create a circuit with a power regulator of 7-40v DC, which opens many ways to power up the raspberry pi.

There it is the link.

Thanks to Morteza Mirmojarabian work, I have a way to you leave your comments... see how here.

Meanwhile I found this two power converters in DealExtreme:

http://dx.com/p/166930 e http://dx.com/p/151211

One more thing to later think a little better about.

Storage expansion

posted 22 Feb 2013, 15:05 by Flip Pipe   [ updated 20 Mar 2013, 02:21 ]

One of the project objectives is to have more storage space. Why? Because compact flash drive space cost 3x more than usb.


Lets compare:

 
 
 
 Sandisk Cruzer Switch USB 2.0 Flash Drive - Black + Red (32GB)
$23.80
$0.74/GB
 SDHC SD Memory Card - Blue (32GB / Class 10)
$25.30
$0.79/GB
 Transcend 400X 32GB Compact Flash
$87.00
$2.71/GB

I think Compact Flash card are used in DSLR because the have a high throughput: 167MB/s, in contrast with 25MB/s of the SHDC (check wikipedia)

Acording DealExtreme, the USB pen I show only writes as 10MB/s (which I think it is the normal in this small size), the SDHC is 14MB7s. They don't have information about the CF card, but if this card was 133x, would have a throughput of 25MB/s... being at 400x, it should read/write a littler fastest.

By the way, USB interface goes up to 60 MB/s.

After the machine empty the buffer to the CF card, this features of the CF cards it is no more necessary, so, why not to add more space, but cheaper.

In Raspberry Pi, there are two options... or a big SC card... or lots of usb flash drives connects through an hub. So, the idea is to join several usb flash drives in a hub:



  or      or else       with this  


After join this pieces, is join all storage space in a raid... with usual HD is simples.... but with usb flash drives will also sem simples with this site:

Then the next step is to choose the right raid level. If I buy 4 usb flash drives of 32GB:


Raid 0; 128Gb and write gains... but if one drive fails, all information goes to garbage.
Raid 5: 96GB  without performance gains... but if one usb flash drive fails, the data (out precious photos) remain intact.
Raid 10: 64GB of storage and double the write speed and I could lose a pen.
Raid 1: This is almost excluded... just 32Gb and no performance gain.

Later, I will do some tests and choose one of them.

Web Server and Wifi

posted 22 Feb 2013, 14:50 by Flip Pipe

This time I find a similar project to BeagleBoard. It is Photo CamCon. In there I find information who to turn the BeagleBoard in an AP and code to web server front end.

Googling a little more ... I found how to do it with raspberry pi and which wifi pen to buy.


Join Gphoto

posted 22 Feb 2013, 14:46 by Flip Pipe

One more link to add to project's bookmarks... This time, the integration gphoto with a Nikon. One idea is to add buttons in order to launch the scripts in the raspberry pi, or in alternative, use the lirc to use an infrared remote control ... more ideas to explore later.

How to shutdown Raspberry

posted 22 Feb 2013, 14:40 by Flip Pipe   [ updated 22 Feb 2013, 14:42 ]

One problem I will have later is shutdown properly the Raspberry Pi, because only remove the power cord should be the last option. Here in this link there is a way to use udev  when unplug a particular USB device it make raspberry pi shutdown.

Later I will see if I find any solutions using GPIO connector.

First Purchases

posted 22 Feb 2013, 14:19 by Flip Pipe


Before starting the project I need to go shopping.

The Raspberry Pi, a company in Oporto (Portugal)...



But since it takes a few more things to connect the Raspberry Pi, I went shopping at DealExtreme:

One memory card to the operating system.


Then a DVI to HDMI converter and a hdmi plan cable to install the operating system.

   

Finally, the energy


And now is wait two or more weeks for this arrive from Hong Kong.


1-9 of 9