# /work quota

**URL:** <https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083>\
**Category:** Unified Model\
**Tags:** ARCHER2\
**Created:** [31 July 2023 08:30 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083 "2023-07-31T08:30:15Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 09:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/23 "2023-08-09T09:58:47Z")

</div>

what do you load to get cdsapi?

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 10:01 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/24 "2023-08-09T10:01:55Z")

</div>

“module load cray-python” - have included it as a pre-script in the definition of HPC\_SERIAL _and_ as an init-script in both HPC\_SERIAL and get\_era\_data

silly question perhaps: it’s definitely running that command on archer, not puma right?

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 10:27 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/25 "2023-08-09T10:27:38Z")

</div>

Oh, I also had to update the /bin/get\_cycle\_offset script to use python3 syntax (for printing $CYCLE\_OFFSET)

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 10:30 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/26 "2023-08-09T10:30:46Z")

</div>

I don’t get cdsapi from cray-python - are you sure you’d not got some other python loaded when it ran on ARCHER ?

```auto
grenvill@ln04:~> module load cray-python
grenvill@ln04:~> module list

Currently Loaded Modules:
  1) craype-x86-rome 5) xpmem/2.5.2-2.4_3.30__gd0f7936.shasta 9) cray-mpich/8.1.23 13) epcc-setup-env
  2) libfabric/1.12.1.2.2.0.0 6) cce/15.0.0 10) cray-libsci/22.12.1.1 14) load-epcc-module
  3) craype-network-ofi 7) craype/2.7.19 11) PrgEnv-cray/8.3.3 15) cray-python/3.9.13.1
  4) perftools-base/22.12.0 8) cray-dsmml/0.2.2 12) bolt/0.8

 

grenvill@ln04:~> python
Python 3.9.13 (main, Aug 10 2022, 17:20:06) 
[GCC 9.3.0 20200312 (Cray Inc.)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdsapi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cdsapi'

```

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 10:36 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/27 "2023-08-09T10:36:21Z")

</div>

That’s super weird.

```auto
shakka@ln03:~> module unload cray-python
shakka@ln03:~> python /work/n02/n02/shakka/download_startfile.py
Traceback (most recent call last):
  File "/work/n02/n02/shakka/download_startfile.py", line 2, in <module>
    import cdsapi
**ImportError: No module named cdsapi**

shakka@ln03:~> module load cray-python
shakka@ln03:~> python /work/n02/n02/shakka/download_startfile.py --cycle 20200304T0000Z --remove 20200303T0000Z
20200304T0000Z
Starting MARS retrieval for 2020030400
2023-08-09 11:35:43,573 INFO Welcome to the CDS
2023-08-09 11:35:43,573 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5.1-complete
2023-08-09 11:35:43,684 INFO Request is queued

```

I can only get cdsapi when cray-python is enabled.

Am going to edit the script and see if it works when I explicitly specify python3…

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 10:51 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/28 "2023-08-09T10:51:24Z")

</div>

what do you get for

```auto
which python

echo $PYTHONPATH

```

`module list`

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 10:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/29 "2023-08-09T10:54:41Z")

</div>

> [@grenville](#):
>
> ```auto
> which python
> 
> echo $PYTHONPATH
> 
> ```
> 
> `module list`

```auto
> /opt/cray/pe/python/3.9.13.1:/work/y07/shared/umshared/lib/python3.9:/work/y07/shared/utils/core/bolt/0.8/modules
> 
> Currently Loaded Modules:
> 1) craype-x86-rome 5) xpmem/2.5.2-2.4_3.30__gd0f7936.shasta 9) cray-mpich/8.1.23 13) epcc-setup-env
> 2) libfabric/1.12.1.2.2.0.0 6) cce/15.0.0 10) cray-libsci/22.12.1.1 14) load-epcc-module
> 3) craype-network-ofi 7) craype/2.7.19 11) PrgEnv-cray/8.3.3 15) cray-python/3.9.13.1
> 4) perftools-base/22.12.0 8) cray-dsmml/0.2.2 12) bolt/0.8

```

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 11:46 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/30 "2023-08-09T11:46:38Z")

</div>

I suspect it’s something to do with python2/3, as I get a python error about the format of the print statement in get\_cycle\_offset if I leave L71 as print cycle\_offset, but if I change to python 3 syntax (ie. print(cycle\_offset) ), I get the following error:

> usage: get\_cycle\_offset [-h] [–curr\_date CURR\_DATE]  
> [–dates DATES [DATES …]]  
> get\_cycle\_offset: error: argument --dates: expected at least one argument

The offending line in suite.rc seems to be:

`CYCLE_OFFSET=$($CYLC_SUITE_RUN_DIR/bin/get_cycle_offset --curr_date $CURR_DATE --dates {{DATES|join(" ")}})`

which I added to get\_era\_data to make $CYCLE\_OFFSET available as an environment variable that I can feed to download\_startfile.py .

Is there another way I can get $CYCLE\_OFFSET to work…?

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 12:10 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/31 "2023-08-09T12:10:55Z")

</div>

Ella

What’s happening here in your .bashrc: export PATH=“/home/n02/n02/shakka/.local/bin:$PATH”

I don’t have permission to see in.  
You probably need to add that to the get\_era\_data pre-script

Grenville

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 12:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/32 "2023-08-09T12:15:33Z")

</div>

Hmm, not sure what’s going on there, but this is the full path:

> /home/n02/n02/shakka/.local/bin:/opt/cray/pe/python/3.9.13.1/bin:/work/n02/n02/shakka:/home/n02/n02/shakka/.local/bin:/work/y07/shared/utils/core/bolt/0.8/bin:/work/y07/shared/utils/core/bin:/opt/cray/pe/mpich/8.1.23/ofi/crayclang/10.0/bin:/opt/cray/pe/mpich/8.1.23/bin:/opt/cray/pe/craype/2.7.19/bin:/opt/cray/pe/cce/15.0.0/binutils/x86\_64/x86\_64-pc-linux-gnu/bin:/opt/cray/pe/cce/15.0.0/binutils/cross/x86\_64-aarch64/aarch64-linux-gnu/…/bin:/opt/cray/pe/cce/15.0.0/utils/x86\_64/bin:/opt/cray/pe/cce/15.0.0/bin:/opt/cray/pe/cce/15.0.0/cce-clang/x86\_64/bin:/opt/cray/pe/perftools/22.12.0/bin:/opt/cray/pe/papi/6.0.0.17/bin:/opt/cray/libfabric/1.12.1.2.2.0.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/opt/cray/pe/bin:/work/y07/shared/umshared/software/bin:/work/y07/shared/umshared/bin:/work/y07/shared/utils/core/python/miniconda2/bin

… So I should add the export PATH=“/home/n02/n02/shakka/.local/bin:$PATH” line to my get\_era\_data pre-script?

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 13:03 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/33 "2023-08-09T13:03:22Z")

</div>

pl allow us to read /home/n02/n02/shakka/.local/bin

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 13:42 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/34 "2023-08-09T13:42:04Z")

</div>

> [@grenville](#):
>
> /home/n02/n02/shakka/.local/bin

Done!

(adding some characters)

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 13:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/35 "2023-08-09T13:55:04Z")

</div>

```auto
grenvill@ln04:/home/n02/n02/dcase/cylc-run/u-cw083> cd /home/n02/n02/shakka/.local/bin
-bash: cd: /home/n02/n02/shakka/.local/bin: Permission denied

chmod -R g+rX /home/n02/n02/shakka (will let us read everything)

```

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 13:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/36 "2023-08-09T13:56:39Z")

</div>

> [@grenville](#):
>
> `chmod -R g+rX /home/n02/n02/shakka`

Strange, thought that’s what I’d done, but evidently not. Should hopefully work now.

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 14:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/37 "2023-08-09T14:57:49Z")

</div>

Well, I’ve failed to get cdsapi - I have used your basrc and bash\_profile

please could you

```auto
python
import cdsapi
print(cdsapi. __file__ )

```

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 15:09 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/38 "2023-08-09T15:09:32Z")

</div>

> [@grenville](#):
>
> ```auto
> python
> import cdsapi
> print(cdsapi. __file__ )
> 
> ```

aha, it’s local: /home/n02/n02/shakka/.local/lib/python3.9/site-packages/cdsapi/ **init**.py

I guess I must’ve installed it within my cray-python environment?

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 16:18 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/39 "2023-08-09T16:18:05Z")

</div>

Ella

You must have done some cdsapi set up - you have hidden config files - I don’t understand your environment.

I got cdsapi by doing

` export PYTHONPATH=/home/n02/n02/shakka/.local/lib/python3.9/site-packages:$PYTHONPATH`

but needed a `.cdsapirc` which I don’t have

try adding this to the pre-script

( or

```auto
export PATH="/home/n02/n02/shakka/.local/bin:$PATH"
export PATH="/work/n02/n02/shakka/:$PATH"

```

but that didn’t work for me, I still can’t see where your current set up adds the module)

Grenville

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 16:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/40 "2023-08-09T16:45:15Z")

</div>

> [@grenville](#):
>
> export PYTHONPATH=/home/n02/n02/shakka/.local/lib/python3.9/site-packages:$PYTHONPATH

Ah ok, I have a .cdsapirc file with my authentication details. I’m now able to run ‘module load cray-python’ it seems, but I’m still getting an error related to the $CYCLE\_OFFSET variable:

```auto
> usage: get_cycle_offset [-h] [--curr_date CURR_DATE]
> [--dates DATES [DATES ...]]
> get_cycle_offset: error: argument --dates: expected at least one argument

```

---

<div class="post-metadata">

**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\
**Post date:** [9 August 2023 17:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/41 "2023-08-09T17:15:41Z")

</div>

I’m guessing {{DATES|join(" ")}} is empty - what are DATES?

---

<div class="post-metadata">

**Author:** ![shakka](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/shakka/32/111_2.png) [@shakka](https://cms-helpdesk.ncas.ac.uk/u/shakka)\
**Post date:** [9 August 2023 17:18 UTC](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083/42 "2023-08-09T17:18:43Z")

</div>

yes they are. It’s the line used in the housekeeping script as a call to get\_cycle\_offset to determine which cycle files to delete. I foolishly assumed I could call it here in the same way to get the current cycle. Interestingly the $CURR\_DATE works as I can print that out to the .out file, so I could potentially do something clever with that in my python script…

[Previous page](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083.md?page=1)

[Next page](https://cms-helpdesk.ncas.ac.uk/t/work-quota/1083.md?page=3)