{ "cells": [ { "cell_type": "markdown", "id": "8f149ebd", "metadata": {}, "source": [ "# How to use" ] }, { "cell_type": "raw", "id": "590da8e3", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "Example file:\n", " \n", ".. literalinclude:: ../tests/example2.csv\n", " :encoding: utf-16" ] }, { "cell_type": "markdown", "id": "9f42acae", "metadata": {}, "source": [ "To read this file:" ] }, { "cell_type": "code", "execution_count": 1, "id": "c614c8c5", "metadata": {}, "outputs": [], "source": [ "import korexo_profile\n", "data = korexo_profile.read(\"../tests/example2.csv\")" ] }, { "cell_type": "markdown", "id": "486c60df", "metadata": {}, "source": [ "This returns a dictionary with three keys:" ] }, { "cell_type": "code", "execution_count": 2, "id": "f3729d25", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['metadata', 'datasets', 'dataframe'])" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.keys()" ] }, { "cell_type": "markdown", "id": "99be49a5", "metadata": {}, "source": [ "Sometimes, people open the raw file in Excel to view or edit it in some small\n", "way and then save as a CSV. This seems innocuous, but completely changes\n", "the format (and usually the encoding also - see below). \n", "\n", "## Raw pandas dataframe\n", "\n", "The raw data is read into a pandas dataframe:" ] }, { "cell_type": "code", "execution_count": 3, "id": "1315ebd8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 9 entries, 0 to 8\n", "Data columns (total 22 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 Date (MM/DD/YYYY) 9 non-null object \n", " 1 Time (HH:mm:ss) 9 non-null object \n", " 2 Time (Fract. Sec) 9 non-null int64 \n", " 3 Site Name 9 non-null object \n", " 4 Cond µS/cm 9 non-null int64 \n", " 5 Depth m 9 non-null float64\n", " 6 nLF Cond µS/cm 9 non-null float64\n", " 7 ODO % sat 9 non-null float64\n", " 8 ODO % local 9 non-null float64\n", " 9 ODO mg/L 9 non-null float64\n", " 10 ORP mV 9 non-null float64\n", " 11 Pressure psi a 9 non-null float64\n", " 12 Sal psu 9 non-null int64 \n", " 13 SpCond µS/cm 9 non-null float64\n", " 14 TDS mg/L 9 non-null int64 \n", " 15 pH 9 non-null float64\n", " 16 pH mV 9 non-null float64\n", " 17 Temp °C 9 non-null float64\n", " 18 Vertical Position m 9 non-null float64\n", " 19 Battery V 9 non-null float64\n", " 20 Cable Pwr V 9 non-null float64\n", " 21 DTW 1 non-null float64\n", "dtypes: float64(15), int64(4), object(3)\n", "memory usage: 1.7+ KB\n" ] } ], "source": [ "df = data[\"dataframe\"]\n", "df.info()" ] }, { "cell_type": "code", "execution_count": 4, "id": "4947ab76", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Date (MM/DD/YYYY)Time (HH:mm:ss)Time (Fract. Sec)Site NameCond µS/cmDepth mnLF Cond µS/cmODO % satODO % localODO mg/L...Sal psuSpCond µS/cmTDS mg/LpHpH mVTemp °CVertical Position mBattery VCable Pwr VDTW
010/16/202015:49:330SQR0973-0.0013.4102.3102.59.50...03.429.23-137.618.996-0.0023.1311.44.2
110/16/202015:49:340SQR0973-0.0013.4102.3102.59.50...03.429.23-137.618.998-0.0013.1311.4NaN
210/16/202015:49:350SQR0973-0.0013.4102.3102.59.49...03.429.23-137.618.999-0.0013.1311.4NaN
310/16/202015:49:360SQR0973-0.0013.4102.3102.59.49...03.429.23-137.619.001-0.0013.1311.4NaN
410/16/202015:49:370SQR0973-0.0013.4102.3102.59.50...03.429.23-137.619.002-0.0013.1311.4NaN
\n", "

5 rows × 22 columns

\n", "
" ], "text/plain": [ " Date (MM/DD/YYYY) Time (HH:mm:ss) Time (Fract. Sec) Site Name Cond µS/cm \\\n", "0 10/16/2020 15:49:33 0 SQR097 3 \n", "1 10/16/2020 15:49:34 0 SQR097 3 \n", "2 10/16/2020 15:49:35 0 SQR097 3 \n", "3 10/16/2020 15:49:36 0 SQR097 3 \n", "4 10/16/2020 15:49:37 0 SQR097 3 \n", "\n", " Depth m nLF Cond µS/cm ODO % sat ODO % local ODO mg/L ... Sal psu \\\n", "0 -0.001 3.4 102.3 102.5 9.50 ... 0 \n", "1 -0.001 3.4 102.3 102.5 9.50 ... 0 \n", "2 -0.001 3.4 102.3 102.5 9.49 ... 0 \n", "3 -0.001 3.4 102.3 102.5 9.49 ... 0 \n", "4 -0.001 3.4 102.3 102.5 9.50 ... 0 \n", "\n", " SpCond µS/cm TDS mg/L pH pH mV Temp °C Vertical Position m \\\n", "0 3.4 2 9.23 -137.6 18.996 -0.002 \n", "1 3.4 2 9.23 -137.6 18.998 -0.001 \n", "2 3.4 2 9.23 -137.6 18.999 -0.001 \n", "3 3.4 2 9.23 -137.6 19.001 -0.001 \n", "4 3.4 2 9.23 -137.6 19.002 -0.001 \n", "\n", " Battery V Cable Pwr V DTW \n", "0 3.13 11.4 4.2 \n", "1 3.13 11.4 NaN \n", "2 3.13 11.4 NaN \n", "3 3.13 11.4 NaN \n", "4 3.13 11.4 NaN \n", "\n", "[5 rows x 22 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "markdown", "id": "9b05c4d8", "metadata": {}, "source": [ "## Per dataset (recommended)\n", "\n", "The information is also stored per-dataset, with some data-type conversions applied under the ``\"datasets\"`` key.\n", "I would recommend using this in most cases. Each item is a dictionary with keys as shown below.\n", "\n", "The Date, Time, and Site Name columns have the handy ``\"median\"`` key, while all others you'd be more interested\n", "in the ``\"data\"`` key which contains a numpy ``ndarray`` with the data:" ] }, { "cell_type": "code", "execution_count": 5, "id": "c8c9f078", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'name': 'Date',\n", " 'column': 'Date (MM/DD/YYYY)',\n", " 'sensor': '',\n", " 'mean': ,\n", " 'stdev': ,\n", " 'data': [datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16),\n", " datetime.date(2020, 10, 16)],\n", " 'median': datetime.date(2020, 10, 16)}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[\"datasets\"][0]" ] }, { "cell_type": "code", "execution_count": 6, "id": "5c5e02e7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'name': 'Time',\n", " 'column': 'Time (HH:mm:ss)',\n", " 'sensor': '',\n", " 'mean': ,\n", " 'stdev': ,\n", " 'data': array(['15:49:33', '15:49:34', '15:49:35', '15:49:36', '15:49:37',\n", " '15:49:38', '15:49:39', '15:49:40', '15:49:41'], dtype=object),\n", " 'median': '15:49:33'}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[\"datasets\"][1]" ] }, { "cell_type": "markdown", "id": "e246a87a", "metadata": {}, "source": [ "And the last one:" ] }, { "cell_type": "code", "execution_count": 7, "id": "9665d50e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'name': 'ODO mg/L',\n", " 'column': 'ODO mg/L',\n", " 'sensor': '19D101830',\n", " 'mean': 7.66,\n", " 'stdev': 0.97,\n", " 'data': array([9.5 , 9.5 , 9.49, 9.49, 9.5 , 9.49, 9.49, 9.5 , 9.49]),\n", " 'median': 9.49}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[\"datasets\"][9]" ] }, { "cell_type": "markdown", "id": "6e266fdf", "metadata": {}, "source": [ "## Other metadata\n", "\n", "Raw file metadata is also available:" ] }, { "cell_type": "code", "execution_count": 8, "id": "d4ffea20", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'created_file': datetime.datetime(2022, 4, 7, 16, 45, 10, 874033),\n", " 'modified_file': datetime.datetime(2022, 4, 7, 16, 45, 10, 875029),\n", " 'created_info': '10/16/2020 6:21:53 AM,,,,,,,,,,,,,,,,,,,,',\n", " 'header_line_no': 9,\n", " 'params': ['Cond µS/cm',\n", " 'Depth m',\n", " 'nLF Cond µS/cm',\n", " 'ODO % sat',\n", " 'ODO % local',\n", " 'ODO mg/L',\n", " 'ORP mV',\n", " 'Pressure psi a',\n", " 'Sal psu',\n", " 'SpCond µS/cm',\n", " 'TDS mg/L',\n", " 'pH',\n", " 'pH mV',\n", " 'Temp °C',\n", " 'Vertical Position m',\n", " 'Battery V',\n", " 'Cable Pwr V',\n", " 'DTW'],\n", " 'sensors': ['19A103955',\n", " '19B104242',\n", " '19A103955',\n", " '19D101830',\n", " '19D101830',\n", " '19D101830',\n", " '19B105042',\n", " '19B104242',\n", " '19A103955',\n", " '19A103955',\n", " '19A103955',\n", " '19B105042',\n", " '19B105042',\n", " '19A103955',\n", " '19B104242',\n", " '19C000969',\n", " '19C000969',\n", " ''],\n", " 'means': [1172.5,\n", " 0.605,\n", " 1277.2,\n", " 85.5,\n", " 85.6,\n", " 7.66,\n", " 47.1,\n", " 0.859,\n", " 0.64,\n", " 1268.1,\n", " 824,\n", " ,\n", " -126,\n", " 20.679,\n", " 0.812,\n", " 3.13,\n", " 11.4,\n", " nan],\n", " 'stdevs': [566.7,\n", " 0.791,\n", " 617.1,\n", " 9.2,\n", " 9.2,\n", " 0.97,\n", " 1.1,\n", " 1.123,\n", " 0.31,\n", " 612.7,\n", " 398,\n", " 0.1,\n", " 5.7,\n", " 0.8,\n", " 0.972,\n", " 0,\n", " 0,\n", " nan]}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data[\"metadata\"]" ] }, { "cell_type": "markdown", "id": "54afcd64", "metadata": {}, "source": [ "## Date formats\n", "\n", "I have found that the way the KorEXO sondes encode dates to be a little messy.\n", "Note that the column header for the first column seems to encode the date\n", "format: ``Date (MM/DD/YYYY)``. This is detected and used automatically by default.\n", "However, I've seen some files produced where the date format is backwards, so\n", "you can also choose to specify the appropriate format in Python datetime \n", "format specifiers:\n", "\n", "For this example:" ] }, { "cell_type": "raw", "id": "a741520f", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. literalinclude:: ../tests/example1.csv\n", " :encoding: utf-16-le" ] }, { "cell_type": "markdown", "id": "0c02cf45", "metadata": {}, "source": [ "The default automatic detection:" ] }, { "cell_type": "code", "execution_count": 9, "id": "44bf7c14", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "datetime.date(2019, 11, 12)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = korexo_profile.read(\"../tests/example1.csv\", datefmt=\"auto\")\n", "data[\"datasets\"][0][\"median\"]" ] }, { "cell_type": "markdown", "id": "19fb9bb3", "metadata": {}, "source": [ "Specifying it directly should provide the same result:" ] }, { "cell_type": "code", "execution_count": 10, "id": "4d089a5a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "datetime.date(2019, 11, 12)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = korexo_profile.read(\"../tests/example1.csv\", datefmt=\"%m/%d/%Y\")\n", "data[\"datasets\"][0][\"median\"]" ] }, { "cell_type": "markdown", "id": "a137ef5d", "metadata": {}, "source": [ "But in this case the true date is actually December 11th, which we can force:" ] }, { "cell_type": "code", "execution_count": 11, "id": "3cad7381", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "datetime.date(2019, 12, 11)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = korexo_profile.read(\"../tests/example1.csv\", datefmt=\"%d/%m/%Y\")\n", "data[\"datasets\"][0][\"median\"]" ] }, { "cell_type": "markdown", "id": "4be718ad", "metadata": {}, "source": [ "## Beware encodings\n", "\n", "I believe the sondes use UTF-16-LE by default. Most text editors do not use\n", "this, so if someone has opened the file and then saved it, it likely will\n", "have converted it to UTF-8 or Windows-1252.\n", "\n", "The default behaviour is simplest:" ] }, { "cell_type": "code", "execution_count": 12, "id": "2ec6ef69", "metadata": {}, "outputs": [], "source": [ "data = korexo_profile.read(\"../tests/example2.csv\")" ] }, { "cell_type": "markdown", "id": "431aefad", "metadata": {}, "source": [ "This is equivalent internally to:" ] }, { "cell_type": "code", "execution_count": 13, "id": "c2f354e7", "metadata": {}, "outputs": [], "source": [ "data = korexo_profile.read(\"../tests/example2.csv\", encoding=\"utf-16\", auto_revert_encoding=\"cp1252\")" ] }, { "cell_type": "markdown", "id": "82f22fcc", "metadata": {}, "source": [ "Because ``auto_revert_encoding`` is not False, the code will first check\n", "whether example2.csv is UTF-16. If it is not, then the file will be loaded\n", "using the value of ``auto_revert_encoding`` (``'cp1252'``). If it is UTF-16,\n", "it will be loaded as such.\n", "\n", "If you want the code to simply use the encoding you direct it to, set\n", "``auto_revert_encoding=False``. This may result in messy errors and weird\n", "behaviour.\n", "\n", "## Skip all that... just give me a complete spreadsheet" ] }, { "cell_type": "code", "execution_count": 14, "id": "14b936ca", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datetimedatetimetime_secsitecondwater_depthcond_nlfdo_satdo_local...presssal_psuspcondtdsphph_mvtempvert_posbatterycable_power
02019-11-12 08:40:302019-11-1208:40:300.0SQR1009.10.00211.598.097.4...0.0020.011.477.59-44.014.3540.0033.0611.1
12019-11-12 08:40:312019-11-1208:40:310.0SQR1009.00.00211.598.097.5...0.0020.011.477.59-43.814.3520.0043.0611.1
22019-11-12 08:40:322019-11-1208:40:320.0SQR1009.00.00211.598.197.6...0.0020.011.377.59-43.714.3490.0003.0611.1
32019-11-12 08:40:332019-11-1208:40:330.0SQR1009.00.00211.598.698.1...0.0030.011.377.61-45.414.3570.0033.0611.1
42019-11-12 08:40:342019-11-1208:40:340.0SQR1009.10.00211.599.398.8...0.0030.011.477.87-59.514.4200.0013.0611.1
\n", "

5 rows × 22 columns

\n", "
" ], "text/plain": [ " datetime date time time_sec site cond \\\n", "0 2019-11-12 08:40:30 2019-11-12 08:40:30 0.0 SQR100 9.1 \n", "1 2019-11-12 08:40:31 2019-11-12 08:40:31 0.0 SQR100 9.0 \n", "2 2019-11-12 08:40:32 2019-11-12 08:40:32 0.0 SQR100 9.0 \n", "3 2019-11-12 08:40:33 2019-11-12 08:40:33 0.0 SQR100 9.0 \n", "4 2019-11-12 08:40:34 2019-11-12 08:40:34 0.0 SQR100 9.1 \n", "\n", " water_depth cond_nlf do_sat do_local ... press sal_psu spcond tds \\\n", "0 0.002 11.5 98.0 97.4 ... 0.002 0.0 11.4 7 \n", "1 0.002 11.5 98.0 97.5 ... 0.002 0.0 11.4 7 \n", "2 0.002 11.5 98.1 97.6 ... 0.002 0.0 11.3 7 \n", "3 0.002 11.5 98.6 98.1 ... 0.003 0.0 11.3 7 \n", "4 0.002 11.5 99.3 98.8 ... 0.003 0.0 11.4 7 \n", "\n", " ph ph_mv temp vert_pos battery cable_power \n", "0 7.59 -44.0 14.354 0.003 3.06 11.1 \n", "1 7.59 -43.8 14.352 0.004 3.06 11.1 \n", "2 7.59 -43.7 14.349 0.000 3.06 11.1 \n", "3 7.61 -45.4 14.357 0.003 3.06 11.1 \n", "4 7.87 -59.5 14.420 0.001 3.06 11.1 \n", "\n", "[5 rows x 22 columns]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = korexo_profile.read(\"../tests/example1_full.csv\")\n", "df = korexo_profile.convert_datasets_to_df(data[\"datasets\"])\n", "df.head()" ] }, { "cell_type": "markdown", "id": "f66e813a", "metadata": {}, "source": [ "And let's say the depth to water was 5.15 m, so let's add that offset: " ] }, { "cell_type": "code", "execution_count": 15, "id": "d38a3831", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datetimedatetimetime_secsitecondwater_depthcond_nlfdo_satdo_local...sal_psuspcondtdsphph_mvtempvert_posbatterycable_powerdtw
02019-11-12 08:40:302019-11-1208:40:300.0SQR1009.10.00211.598.097.4...0.011.477.59-44.014.3540.0033.0611.15.152
12019-11-12 08:40:312019-11-1208:40:310.0SQR1009.00.00211.598.097.5...0.011.477.59-43.814.3520.0043.0611.15.152
22019-11-12 08:40:322019-11-1208:40:320.0SQR1009.00.00211.598.197.6...0.011.377.59-43.714.3490.0003.0611.15.152
32019-11-12 08:40:332019-11-1208:40:330.0SQR1009.00.00211.598.698.1...0.011.377.61-45.414.3570.0033.0611.15.152
42019-11-12 08:40:342019-11-1208:40:340.0SQR1009.10.00211.599.398.8...0.011.477.87-59.514.4200.0013.0611.15.152
\n", "

5 rows × 23 columns

\n", "
" ], "text/plain": [ " datetime date time time_sec site cond \\\n", "0 2019-11-12 08:40:30 2019-11-12 08:40:30 0.0 SQR100 9.1 \n", "1 2019-11-12 08:40:31 2019-11-12 08:40:31 0.0 SQR100 9.0 \n", "2 2019-11-12 08:40:32 2019-11-12 08:40:32 0.0 SQR100 9.0 \n", "3 2019-11-12 08:40:33 2019-11-12 08:40:33 0.0 SQR100 9.0 \n", "4 2019-11-12 08:40:34 2019-11-12 08:40:34 0.0 SQR100 9.1 \n", "\n", " water_depth cond_nlf do_sat do_local ... sal_psu spcond tds ph \\\n", "0 0.002 11.5 98.0 97.4 ... 0.0 11.4 7 7.59 \n", "1 0.002 11.5 98.0 97.5 ... 0.0 11.4 7 7.59 \n", "2 0.002 11.5 98.1 97.6 ... 0.0 11.3 7 7.59 \n", "3 0.002 11.5 98.6 98.1 ... 0.0 11.3 7 7.61 \n", "4 0.002 11.5 99.3 98.8 ... 0.0 11.4 7 7.87 \n", "\n", " ph_mv temp vert_pos battery cable_power dtw \n", "0 -44.0 14.354 0.003 3.06 11.1 5.152 \n", "1 -43.8 14.352 0.004 3.06 11.1 5.152 \n", "2 -43.7 14.349 0.000 3.06 11.1 5.152 \n", "3 -45.4 14.357 0.003 3.06 11.1 5.152 \n", "4 -59.5 14.420 0.001 3.06 11.1 5.152 \n", "\n", "[5 rows x 23 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"dtw\"] = df.water_depth + 5.15\n", "df.head()" ] }, { "cell_type": "markdown", "id": "4040374b", "metadata": {}, "source": [ "Then re-index so the profile is at 5 cm intervals:" ] }, { "cell_type": "code", "execution_count": 16, "id": "c0a73db1", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
time_seccondwater_depthcond_nlfdo_satdo_localdo_concorp_mvpresssal_psuspcondtdsphph_mvtempvert_posbatterycable_power
dtw
5.1500000.09.5333330.00000011.50000091.70000091.2000008.910000-39.2000000.0010000.00000011.3666677.0000008.390000-89.13333316.6636670.0013333.0611.2
5.2021740.01203.0975160.0521741324.40621192.24037391.6701868.264037-42.3807450.0739630.6600001313.976398854.1925478.364037-88.66149120.5838070.4199383.0611.2
5.2543480.01220.5797100.1043481343.95048391.72584591.2258458.212585-41.2033820.1485390.6700001333.350483866.4830928.305169-85.65845420.5730340.6945483.0611.2
5.3065220.01225.0723470.1565221348.89487291.59590591.0959058.199795-40.5897610.2220540.6702051338.192824870.1842958.289795-84.58361820.5751230.8072703.0611.2
5.3586960.01225.9878000.2086961349.88603991.58213891.0821388.199107-40.5553460.2961850.6708931339.177108870.8037748.289107-84.52855320.5755360.8115373.0611.2
\n", "
" ], "text/plain": [ " time_sec cond water_depth cond_nlf do_sat \\\n", "dtw \n", "5.150000 0.0 9.533333 0.000000 11.500000 91.700000 \n", "5.202174 0.0 1203.097516 0.052174 1324.406211 92.240373 \n", "5.254348 0.0 1220.579710 0.104348 1343.950483 91.725845 \n", "5.306522 0.0 1225.072347 0.156522 1348.894872 91.595905 \n", "5.358696 0.0 1225.987800 0.208696 1349.886039 91.582138 \n", "\n", " do_local do_conc orp_mv press sal_psu spcond \\\n", "dtw \n", "5.150000 91.200000 8.910000 -39.200000 0.001000 0.000000 11.366667 \n", "5.202174 91.670186 8.264037 -42.380745 0.073963 0.660000 1313.976398 \n", "5.254348 91.225845 8.212585 -41.203382 0.148539 0.670000 1333.350483 \n", "5.306522 91.095905 8.199795 -40.589761 0.222054 0.670205 1338.192824 \n", "5.358696 91.082138 8.199107 -40.555346 0.296185 0.670893 1339.177108 \n", "\n", " tds ph ph_mv temp vert_pos battery \\\n", "dtw \n", "5.150000 7.000000 8.390000 -89.133333 16.663667 0.001333 3.06 \n", "5.202174 854.192547 8.364037 -88.661491 20.583807 0.419938 3.06 \n", "5.254348 866.483092 8.305169 -85.658454 20.573034 0.694548 3.06 \n", "5.306522 870.184295 8.289795 -84.583618 20.575123 0.807270 3.06 \n", "5.358696 870.803774 8.289107 -84.528553 20.575536 0.811537 3.06 \n", "\n", " cable_power \n", "dtw \n", "5.150000 11.2 \n", "5.202174 11.2 \n", "5.254348 11.2 \n", "5.306522 11.2 \n", "5.358696 11.2 " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df2 = korexo_profile.make_regularly_spaced(df, \"dtw\", step=0.05)\n", "df2.head()" ] } ], "metadata": { "celltoolbar": "Raw Cell Format", "kernelspec": { "display_name": "Python [conda env:wrap]", "language": "python", "name": "conda-env-wrap-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.2" } }, "nbformat": 4, "nbformat_minor": 5 }