new datasets appear in database/files/000/

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

new datasets appear in database/files/000/

Jochen Bick
Hi,

today we found out that some of our files in your galaxy instance are
new in "database/files/000/" which is the first folder of your database
files. Could some please explain me how galaxy generates new file names?
and how this is possible?

Thanks in advance Jochen

--
ETH Zurich
*Jochen Bick*
Animal Physiology
Institute of Agricultural Sciences
Postal address: Universitätstrasse 2 / LFW B 58.1
8092 Zurich, Switzerland
Office: Eschikon 27
8315 Lindau, Switzerland

Phone +41 52 354 92 06
[hidden email] <mailto:[hidden email]>
www.ap.ethz.ch

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: new datasets appear in database/files/000/

Jochen Bick
Coming back to my question:

this is how an ls -ls looks in my database folder...

2. Apr 14:40 000/
9. Nov 2015 005/
19. Nov 2018 007/
19. Nov 2018 008/
3. Dez 2015 010/
10. Jan 2017 012/
10. Jan 2017 013/
29. Jun 2017 014/
4. Jul 2017 015/
9. Jan 2018 018/
29. Jun 2017 020/
21. Sep 2016 021/
20. Mär 2018 032/
29. Jan 09:38 033/
29. Jan 10:32 034/
16. Jul 2018 035/
30. Okt 2018 036/
30. Okt 2018 037/
30. Okt 2018 038/
30. Okt 2018 039/
29. Aug 2018 070/
29. Aug 2018 071/
29. Aug 2018 072/
29. Aug 2018 073/
29. Aug 2018 074/
29. Aug 2018 075/
29. Aug 2018 076/
29. Aug 2018 077/
29. Aug 2018 078/
29. Aug 2018 079/
29. Aug 2018 080/
29. Aug 2018 081/
29. Aug 2018 082/
29. Aug 2018 083/
29. Aug 2018 084/
29. Aug 2018 085/
29. Aug 2018 086/
29. Aug 2018 087/
29. Aug 2018 088/
29. Aug 2018 089/
29. Aug 2018 090/
29. Aug 2018 091/
29. Aug 2018 092/
29. Aug 2018 093/
29. Aug 2018 094/
29. Aug 2018 095/
31. Aug 2018 096/
30. Aug 2018 097/
1. Sep 2018 098/
24. Okt 2018 099/


if you look into the 000 folder it restarts:

10. Dez 19:36 100/
30. Nov 10:45 101/
10. Dez 19:39 102/
10. Dez 19:37 103/
11. Jan 15:13 104/
11. Jan 11:31 105/
22. Jan 14:33 106/
8. Feb 10:46 107/
14. Feb 15:47 108/
21. Mär 11:00 109/
2. Apr 15:22 110/
21. Mai 17:17 111/

is this the expected behavior?

The only thing we did was we archived some of the very early generated
datasets. Is this maybe the problem?


Cheers Jochen


On 14.05.19 15:32, Jochen Bick wrote:
> Hi,
>
> today we found out that some of our files in your galaxy instance are
> new in "database/files/000/" which is the first folder of your database
> files. Could some please explain me how galaxy generates new file names?
> and how this is possible?
>
> Thanks in advance Jochen
>
--
ETH Zurich
*Jochen Bick*
Animal Physiology
Institute of Agricultural Sciences
Postal address: Universitätstrasse 2 / LFW B 58.1
8092 Zurich, Switzerland
Office: Eschikon 27
8315 Lindau, Switzerland

Phone +41 52 354 92 06
[hidden email] <mailto:[hidden email]>
www.ap.ethz.ch

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: new datasets appear in database/files/000/

Dannon Baker-2
Hi Jochen,

Yes, those new directories would coincide with when you passed 100,000
datasets on your instance, so congrats!  You can see the code that builds
this hashed path here:
https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/util/__init__.py#L137

So, for example, dataset id `99999` Will be in folder 'datasets/099/'.
Dataset '100000' will be in 'datasets/000/100/'.

Sorry for the slow response, hope this clears up the storage path logic!

-Dannon

On Tue, May 21, 2019 at 11:38 AM Jochen Bick <[hidden email]>
wrote:

> Coming back to my question:
>
> this is how an ls -ls looks in my database folder...
>
> 2. Apr 14:40 000/
> 9. Nov 2015 005/
> 19. Nov 2018 007/
> 19. Nov 2018 008/
> 3. Dez 2015 010/
> 10. Jan 2017 012/
> 10. Jan 2017 013/
> 29. Jun 2017 014/
> 4. Jul 2017 015/
> 9. Jan 2018 018/
> 29. Jun 2017 020/
> 21. Sep 2016 021/
> 20. Mär 2018 032/
> 29. Jan 09:38 033/
> 29. Jan 10:32 034/
> 16. Jul 2018 035/
> 30. Okt 2018 036/
> 30. Okt 2018 037/
> 30. Okt 2018 038/
> 30. Okt 2018 039/
> 29. Aug 2018 070/
> 29. Aug 2018 071/
> 29. Aug 2018 072/
> 29. Aug 2018 073/
> 29. Aug 2018 074/
> 29. Aug 2018 075/
> 29. Aug 2018 076/
> 29. Aug 2018 077/
> 29. Aug 2018 078/
> 29. Aug 2018 079/
> 29. Aug 2018 080/
> 29. Aug 2018 081/
> 29. Aug 2018 082/
> 29. Aug 2018 083/
> 29. Aug 2018 084/
> 29. Aug 2018 085/
> 29. Aug 2018 086/
> 29. Aug 2018 087/
> 29. Aug 2018 088/
> 29. Aug 2018 089/
> 29. Aug 2018 090/
> 29. Aug 2018 091/
> 29. Aug 2018 092/
> 29. Aug 2018 093/
> 29. Aug 2018 094/
> 29. Aug 2018 095/
> 31. Aug 2018 096/
> 30. Aug 2018 097/
> 1. Sep 2018 098/
> 24. Okt 2018 099/
>
>
> if you look into the 000 folder it restarts:
>
> 10. Dez 19:36 100/
> 30. Nov 10:45 101/
> 10. Dez 19:39 102/
> 10. Dez 19:37 103/
> 11. Jan 15:13 104/
> 11. Jan 11:31 105/
> 22. Jan 14:33 106/
> 8. Feb 10:46 107/
> 14. Feb 15:47 108/
> 21. Mär 11:00 109/
> 2. Apr 15:22 110/
> 21. Mai 17:17 111/
>
> is this the expected behavior?
>
> The only thing we did was we archived some of the very early generated
> datasets. Is this maybe the problem?
>
>
> Cheers Jochen
>
>
> On 14.05.19 15:32, Jochen Bick wrote:
> > Hi,
> >
> > today we found out that some of our files in your galaxy instance are
> > new in "database/files/000/" which is the first folder of your database
> > files. Could some please explain me how galaxy generates new file names?
> > and how this is possible?
> >
> > Thanks in advance Jochen
> >
> --
> ETH Zurich
> *Jochen Bick*
> Animal Physiology
> Institute of Agricultural Sciences
> Postal address: Universitätstrasse 2 / LFW B 58.1
> 8092 Zurich, Switzerland
> Office: Eschikon 27
> 8315 Lindau, Switzerland
>
> Phone +41 52 354 92 06
> [hidden email] <mailto:[hidden email]>
> www.ap.ethz.ch
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/