Loading ...
Sorry, an error occurred while loading the content.

Re: [nslu2-linux] Re: svn_1.5.1-1_arm.ipk

Expand Messages
  • Rod Whitby
    Is Linux using swap to external disk while the svn operation is happening? If the act of checking in large files (and svn trying to do deltas between large
    Message 1 of 8 , Aug 24, 2008
    • 0 Attachment
      Is Linux using swap to external disk while the svn operation is happening?
      If the act of checking in large files (and svn trying to do deltas between large files) causes the memory requirements to go into swap, then that could explain the low speed.
      What speed do you get on an equivalent size transfer, but broken up into many small (say 4k) files?
      -- Rod

      -----Original Message-----
      From: "faz0222" <faz0222@...>
      Date: Sunday, Aug 24, 2008 5:19 pm
      Subject: [nslu2-linux] Re: svn_1.5.1-1_arm.ipk
      To: nslu2-linux@yahoogroups.comReply-To: nslu2-linux@yahoogroups.com

      Hi Brian,
      >
      >I do understand that svn is not designed for "file transfer" but
      >having it transfer at 150k/s over the LAN sounds really extreme to me.
      >
      >> * Does the slowness only happen when you do svn import, or is
      > day-to-day check-in check-out of text file changes also slow?
      >
      >It is slow on import, export and all daily operations.
      >I'm not using it for text, only binaries like image files, so I'm not doing any dif or anything like that, only simple import, update,
      >export, thats about it.
      >
      >I've done a test importing 4 .txt files, ~7MB each
      >The transfer rate was ~60kbyte/s !!
      >
      >> * What is the size of your import? Mostly text files or binary files?
      >
      >Tests were done with ~6 binary tga files for a total of 64MB.
      >It takes ~5 sec for a normal Win Explorer/FTP/SFTP transfer to the NAS and more than 3 minutes for a SVN import.
      >
      >> If you start with small repo with only text files, and try a couple of
      > different size & text/binary combinations, do you see any pattern of
      > the speed?
      >
      >I've first imported text files (60kbytes/s) then imported text and
      >binary files. The speed was as follow :
      >- tga binaries : 160kbytes/s in average for the 1st and 2nd tga files, then for the other tga files the speed goes like this during the
      >transfer of each file 150 > 0 > 180 > 0 > 170 > 0 ...... etc
      >- txt files : ~60kbytes/s
      >
      >I've tested to import tga binary files only and the speed pattern was the same, going from ~150 to 0 over and over starting with the 3rd file.
      >
      >I've also tried to only use "anon-access = write" and no "auth-access" or "authz-db" - same low transfer speed.
      >
      >> Try "sha256sum all-files-to-import" on your NAS and see how long it would take.
      >
      >I've run a cksum on all 6 binary files and it was done in less than a second for each.
      >
      >> What you want to understand, and I'm curious as well, is what is
      > supposed to be the top speed of svnserver on such device.
      >
      >I don't have the answer for that but I would expect it to be closer to the transfer rate I'm getting with FTP/SFTP/Win explorer from computer to NAS, even if its a little bit slower I would understand, but the
      >speed right now is just ridiculous ... :(
      >
      >Thanks
      >
      >--- In nslu2-linux@yahoogroups.com, "Brian Zhou" <b88zhou@...> wrote:
      >
      >> First, I agree that hardware is not likely to be the problem here.
      >
      > I've responded to you in QNAP, but to re-iterate, svn is not designed
      > for file transfer.
      >
      > * Does the slowness only happen when you do svn import, or is
      > day-to-day check-in check-out of text file changes also slow?
      >
      > * What is the size of your import? Mostly text files or binary files?
      > If you start with small repo with only text files, and try a couple of
      > different size & text/binary combinations, do you see any pattern of
      > the speed?
      >
      > What you want to understand, and I'm curious as well, is what is
      > supposed to be the top speed of svnserver on such device. Your device
      > has ARM CPU without hardware FPU, so if svn throws a lot of
      > calculation at it, the high CPU utilization will definitely slow down
      > the network transfer. QNAP libc is already using ARM EABI, it gives
      > you the best FP performance without hardware FPU.
      >
      > Based on my experience, if there's cross compilation or packaging
      > issue, program usually crashes as a result. When it works in the end
      > but takes a long time and high CPU utilization, very likely the
      > upstream program is not designed for the use scenario and no-FPU
      > environment.
      >
      > I don't know the internal of svn, but I can imagine that it needs to
      > calculate the checksum/hash of files during import, so later it can
      > easily figure out whether there is a change. The calculation of
      > checksum/hash alone is CPU intensive. Try "sha256sum
      > all-files-to-import" on your NAS and see how long it would take.
      >
      > -Brian
      >
      > --- In nslu2-linux@yahoogroups.com, "Anthony F." <faz0222@> wrote:
      > >
      > > Hi all,
      > >
      > > I found this group contact in svn_1.5.1-1_arm.ipk that I have
      > installed on
      > > my NAS.
      > >
      > > I've been trying to use svn on my NAS (QNAP-209II) but met some big
      > problems
      > > with the transfer speed.
      > > I've searched for help on both svn and Qnap forums as you can read
      > here :
      > > http://forum.qnap.com/viewtopic.php?f=130&t=6956&st=0&sk=t&sd=a
      > > http://forum.qnap.com/viewtopic.php?f=85&t=7175
      > >
      >
      >http://www.svnforum.org/2017/viewtopic.php?t=6469&sid=ebf3ce4fd13b397182e87710a49059db
      > >
      > > To keep it short, I'm using a QNAP-209II connected through a LAN to
      > a single
      > > pc using XP SP2.
      > > If I transfer files from my pc to the NAS using Win Explorer, SFTP
      > or FTP
      > > the transfer speed are normal.
      > > But if I import files from my pc to SVN on the NAS the transfer
      >rate is
      > > extremely slow (~150k/s) and the NAS cpu is maxed out (99%).
      > >
      > > People have been trying to help me on both forums and they seem to agree
      > > that it's not a hardware problem since I can transfer files
      > correctly if I'm
      > > not using svn. I've also tested 2 different clients (Tortoise and
      > RapidSVN)
      > > and met the same speed issue.
      > >
      > > Any idea what could be the problem ?
      > > Any help would be really appreciated :)
      > >
      > > Thanks in advance and please let me know if there's additional
      > information I
      > > can provide !
      > > Anthony
      > >
      >
      >
      >
      >
      >------------------------------------
      >
      >Yahoo! Groups Links
      >
      >
      >
      >
    Your message has been successfully submitted and would be delivered to recipients shortly.