Compare commits
11 Commits
73468e84b6
...
v0.2.1
| Author | SHA1 | Date | |
|---|---|---|---|
| fede3320c8 | |||
|
4519065a15
|
|||
| b06bd0abcd | |||
|
0201962725
|
|||
|
a0750e4d5f
|
|||
|
ed2ba07ce1
|
|||
|
fe468a528d
|
|||
| c0eb798788 | |||
|
3edb93fcf7
|
|||
|
2e96d174f2
|
|||
|
3ea4d31fd0
|
@@ -39,6 +39,12 @@ jobs:
|
||||
- name: Run tests
|
||||
run: go test ./...
|
||||
|
||||
- name: Run vet
|
||||
run: go vet ./...
|
||||
|
||||
- name: Check reachable vulnerabilities
|
||||
run: go run golang.org/x/vuln/cmd/govulncheck@v1.6.0 ./...
|
||||
|
||||
- name: Build
|
||||
run: go build ./cmd/dataxl
|
||||
|
||||
|
||||
@@ -30,6 +30,12 @@ jobs:
|
||||
- name: Run tests
|
||||
run: go test ./...
|
||||
|
||||
- name: Run vet
|
||||
run: go vet ./...
|
||||
|
||||
- name: Check reachable vulnerabilities
|
||||
run: go run golang.org/x/vuln/cmd/govulncheck@v1.6.0 ./...
|
||||
|
||||
- name: Build release archives
|
||||
env:
|
||||
RELEASE_TAG: ${{ inputs.tag }}
|
||||
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Yuya KAMATAKI and contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
51
README.md
51
README.md
@@ -18,7 +18,7 @@ TSVへ変換したり、Excelからコピーした表を再び構造化データ
|
||||
|
||||
## インストール
|
||||
|
||||
Go 1.24以上が必要です。
|
||||
Go 1.25以上が必要です。
|
||||
|
||||
```sh
|
||||
go install git.rumginger.org/agent/dataxl/cmd/dataxl@latest
|
||||
@@ -111,6 +111,45 @@ id items[0].qty items[0].sku user.name
|
||||
逆方向の変換では、`user.name` や `items[0].sku` のような列名から入れ子の
|
||||
map/arrayを復元します。
|
||||
|
||||
列パスは `.` でmapのキー、`[n]` で0始まりの配列indexを表します。たとえば
|
||||
`orders[0].items[1].sku` は、最初の注文に含まれる2番目の商品の `sku` です。
|
||||
|
||||
`.`、`[`、`]` を含むmapキーや空のmapキーはJSON文字列を使ったbracket記法で
|
||||
表します。たとえば `{"build.target": {"x[y]": 1}}` は
|
||||
`["build.target"]["x[y]"]` という列名になります。
|
||||
|
||||
同じ行に `user` と `user.name` のような競合する列がある場合や、同名ヘッダーが
|
||||
複数ある場合はエラーにします。どちらかの値だけを採用して正常終了することは
|
||||
ありません。
|
||||
|
||||
## セル値の型推定
|
||||
|
||||
CSV、TSV、XLSXから構造化形式へ戻す際は、セル文字列を次の順で推定します。
|
||||
|
||||
- 空文字は空文字列
|
||||
- `true` / `false` はboolean
|
||||
- 通常の整数は64-bit integer
|
||||
- 小数点または指数表記を含む数値は64-bit floating point
|
||||
- それ以外は文字列
|
||||
|
||||
郵便番号や商品コードを想定し、`00123` や `-01` のようなゼロ埋め値は文字列のまま
|
||||
保持します。セルの前後空白も文字列の一部として保持し、` 42 ` のような値を数値に
|
||||
変換しません。日付、時刻、`null` は自動推定しません。
|
||||
|
||||
## データ欠落を防ぐ検証
|
||||
|
||||
- JSON入力はUTF-8として検証し、1つの値と後続空白だけを許可します。2つ目の値、
|
||||
不正な後続データ、object内の重複キーは、値を置換・無視せずエラーにします。
|
||||
- YAML streamに複数文書がある場合は、順序を保った配列としてすべて変換します。
|
||||
- UTF-8 BOM付きCSV/TSVでは先頭のBOMをencoding markerとして除去します。
|
||||
- CSV/TSVはUTF-8として検証し、ヘッダーより列数が多い行をエラーにします。
|
||||
- 空ヘッダー列に値がある場合、重複ヘッダー、復元時に型が競合する列パスは
|
||||
エラーにします。
|
||||
- scalar fieldを1つも持たないrecordは表で表現できないため、空行へ変換して
|
||||
record数を失う代わりにエラーにします。空のrecord listは空の表へ変換できます。
|
||||
- `rows`、`records`、`items` を行配列として展開するのは、それがオブジェクト唯一の
|
||||
キーである場合だけです。同階層のmetadataを破棄しません。
|
||||
|
||||
## CLIオプション
|
||||
|
||||
- `-i`: 入力ファイル。省略時はstdin。
|
||||
@@ -119,15 +158,25 @@ map/arrayを復元します。
|
||||
- `-to`: 出力形式。`json`, `yaml`, `toml`, `csv`, `tsv`, `xlsx`。
|
||||
- `-sheet`: XLSXの読み書きに使うシート名。既定値は `Sheet1`。
|
||||
- `-pretty`: JSONなどの構造化出力を整形するか。既定値は `true`。
|
||||
- `-version`: バージョンを表示して終了します。
|
||||
|
||||
## 現在の制約
|
||||
|
||||
- 表形式では1行目をヘッダーとして扱います。
|
||||
- XLSXは指定した1シートのみ読み書きします。
|
||||
- セル値の型推定は、空文字、真偽値、整数、小数、文字列の範囲です。
|
||||
- 空のmap/arrayは表側で `{}` / `[]` と表示されますが、逆変換時は文字列になります。
|
||||
- structured形式の `null` と表形式の空文字は同じ空セルになります。
|
||||
- top-level scalarやscalar配列は表側で `value` 列を使うため、逆変換時は
|
||||
`value` キーを持つレコードになります。
|
||||
- 複雑なExcel書式や数式の保持は目的外です。
|
||||
|
||||
## 開発者向け情報
|
||||
|
||||
- 設計概要: [docs/architecture.md](docs/architecture.md)
|
||||
- 開発手順: [docs/development.md](docs/development.md)
|
||||
|
||||
## License
|
||||
|
||||
dataxlは[MIT License](LICENSE)で提供します。配布バイナリに含まれる依存ソフトウェアの
|
||||
著作権表示とライセンスは[THIRD_PARTY_LICENSES.txt](THIRD_PARTY_LICENSES.txt)を参照してください。
|
||||
|
||||
764
THIRD_PARTY_LICENSES.txt
Normal file
764
THIRD_PARTY_LICENSES.txt
Normal file
@@ -0,0 +1,764 @@
|
||||
THIRD-PARTY SOFTWARE NOTICES
|
||||
|
||||
This file contains license notices for third-party software included in the distributed binary.
|
||||
|
||||
================================================================================
|
||||
Package: Go standard library
|
||||
License: BSD-3-Clause
|
||||
Source: https://go.dev/LICENSE
|
||||
|
||||
Copyright 2009 The Go Authors.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google LLC nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/BurntSushi/toml
|
||||
License: MIT
|
||||
Source: https://github.com/BurntSushi/toml/blob/v1.6.0/COPYING
|
||||
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) 2013 TOML authors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/richardlehane/mscfb
|
||||
License: Apache-2.0
|
||||
Source: https://github.com/richardlehane/mscfb/blob/v1.0.7/LICENSE.txt
|
||||
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/richardlehane/msoleps/types
|
||||
License: Apache-2.0
|
||||
Source: https://github.com/richardlehane/msoleps/blob/v1.0.6/LICENSE.txt
|
||||
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/tiendc/go-deepcopy
|
||||
License: MIT
|
||||
Source: https://github.com/tiendc/go-deepcopy/blob/v1.7.2/LICENSE
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2023 tiendc
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/xuri/efp
|
||||
License: BSD-3-Clause
|
||||
Source: https://github.com/xuri/efp/blob/v0.0.1/LICENSE
|
||||
|
||||
BSD 3-Clause License
|
||||
|
||||
Copyright (c) 2017 - 2025 Ri Xu All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright notice, this
|
||||
list of conditions and the following disclaimer.
|
||||
|
||||
* Redistributions in binary form must reproduce the above copyright notice,
|
||||
this list of conditions and the following disclaimer in the documentation
|
||||
and/or other materials provided with the distribution.
|
||||
|
||||
* Neither the name of efp nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
||||
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
||||
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
||||
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/xuri/excelize/v2
|
||||
License: BSD-3-Clause
|
||||
Source: https://github.com/xuri/excelize/blob/v2.11.0/LICENSE
|
||||
|
||||
BSD 3-Clause License
|
||||
|
||||
Copyright (c) 2016-2026 The excelize Authors.
|
||||
Copyright (c) 2011-2017 Geoffrey J. Teale
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright notice, this
|
||||
list of conditions and the following disclaimer.
|
||||
|
||||
* Redistributions in binary form must reproduce the above copyright notice,
|
||||
this list of conditions and the following disclaimer in the documentation
|
||||
and/or other materials provided with the distribution.
|
||||
|
||||
* Neither the name of the copyright holder nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
||||
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
||||
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
||||
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
================================================================================
|
||||
Package: github.com/xuri/nfp
|
||||
License: BSD-3-Clause
|
||||
Source: https://github.com/xuri/nfp/blob/2ddeb826f9a9/LICENSE
|
||||
|
||||
BSD 3-Clause License
|
||||
|
||||
Copyright (c) 2022-2025 Ri Xu All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright notice, this
|
||||
list of conditions and the following disclaimer.
|
||||
|
||||
* Redistributions in binary form must reproduce the above copyright notice,
|
||||
this list of conditions and the following disclaimer in the documentation
|
||||
and/or other materials provided with the distribution.
|
||||
|
||||
* Neither the name of nfp nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
||||
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
||||
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
||||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
||||
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
================================================================================
|
||||
Package: golang.org/x/crypto
|
||||
License: BSD-3-Clause
|
||||
Source: https://cs.opensource.google/go/x/crypto/+/v0.53.0:LICENSE
|
||||
|
||||
Copyright 2009 The Go Authors.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google LLC nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
================================================================================
|
||||
Package: golang.org/x/net/html
|
||||
License: BSD-3-Clause
|
||||
Source: https://cs.opensource.google/go/x/net/+/v0.56.0:LICENSE
|
||||
|
||||
Copyright 2009 The Go Authors.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google LLC nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
================================================================================
|
||||
Package: golang.org/x/text
|
||||
License: BSD-3-Clause
|
||||
Source: https://cs.opensource.google/go/x/text/+/v0.39.0:LICENSE
|
||||
|
||||
Copyright 2009 The Go Authors.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google LLC nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
================================================================================
|
||||
Package: gopkg.in/yaml.v3
|
||||
License: MIT
|
||||
Source: https://github.com/go-yaml/yaml/blob/v3.0.1/LICENSE
|
||||
|
||||
|
||||
This project is covered by two different licenses: MIT and Apache.
|
||||
|
||||
#### MIT License ####
|
||||
|
||||
The following files were ported to Go from C files of libyaml, and thus
|
||||
are still covered by their original MIT license, with the additional
|
||||
copyright staring in 2011 when the project was ported over:
|
||||
|
||||
apic.go emitterc.go parserc.go readerc.go scannerc.go
|
||||
writerc.go yamlh.go yamlprivateh.go
|
||||
|
||||
Copyright (c) 2006-2010 Kirill Simonov
|
||||
Copyright (c) 2006-2011 Kirill Simonov
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
### Apache License ###
|
||||
|
||||
All the remaining project files are covered by the Apache license:
|
||||
|
||||
Copyright (c) 2011-2019 Canonical Ltd
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
79
cmd/dataxl/conversion.go
Normal file
79
cmd/dataxl/conversion.go
Normal file
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// convert routes data through either the structured-value or table internal
|
||||
// representation. Keeping this matrix in one place makes format adapters
|
||||
// independent from CLI and file I/O concerns.
|
||||
func convert(input []byte, from, to, sheet string, pretty bool) ([]byte, error) {
|
||||
switch {
|
||||
case isTabular(from) && isTabular(to):
|
||||
t, err := parseTable(input, from, sheet)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return encodeTable(t, to, sheet)
|
||||
case isTabular(from):
|
||||
t, err := parseTable(input, from, sheet)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
records, err := tableToRecords(t)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return encodeStructured(records, to, pretty)
|
||||
case isTabular(to):
|
||||
value, err := parseStructured(input, from)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
t := valueToTable(value)
|
||||
if len(t.Header) == 0 && len(t.Rows) > 0 {
|
||||
return nil, fmt.Errorf("structured value has records but no scalar fields to represent as a table")
|
||||
}
|
||||
return encodeTable(t, to, sheet)
|
||||
default:
|
||||
value, err := parseStructured(input, from)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return encodeStructured(value, to, pretty)
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeFormat(format string) string {
|
||||
format = strings.ToLower(strings.TrimPrefix(strings.TrimSpace(format), "."))
|
||||
switch format {
|
||||
case "yml":
|
||||
return "yaml"
|
||||
case "xlsm", "xls":
|
||||
return "xlsx"
|
||||
default:
|
||||
return format
|
||||
}
|
||||
}
|
||||
|
||||
func inferFormat(path string) string {
|
||||
if path == "" || path == "-" {
|
||||
return ""
|
||||
}
|
||||
return normalizeFormat(filepath.Ext(path))
|
||||
}
|
||||
|
||||
func validateFormat(format string) error {
|
||||
switch format {
|
||||
case "json", "yaml", "toml", "csv", "tsv", "xlsx":
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("unsupported format %q", format)
|
||||
}
|
||||
}
|
||||
|
||||
func isTabular(format string) bool {
|
||||
return format == "csv" || format == "tsv" || format == "xlsx"
|
||||
}
|
||||
64
cmd/dataxl/conversion_matrix_test.go
Normal file
64
cmd/dataxl/conversion_matrix_test.go
Normal file
@@ -0,0 +1,64 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestConversionMatrixPreservesRepresentativeTable(t *testing.T) {
|
||||
formats := []string{"json", "yaml", "toml", "csv", "tsv", "xlsx"}
|
||||
records := []any{
|
||||
map[string]any{
|
||||
"active": true,
|
||||
"id": int64(1),
|
||||
"name": "Alice",
|
||||
"note": " first line\nsecond line ",
|
||||
},
|
||||
map[string]any{
|
||||
"active": false,
|
||||
"id": int64(2),
|
||||
"name": "ボブ",
|
||||
"note": "comma, tab\t and quote \"",
|
||||
},
|
||||
}
|
||||
want := valueToTable(records)
|
||||
|
||||
fixtures := make(map[string][]byte, len(formats))
|
||||
for _, format := range formats {
|
||||
var (
|
||||
data []byte
|
||||
err error
|
||||
)
|
||||
if isTabular(format) {
|
||||
data, err = encodeTable(want, format, "Sheet1")
|
||||
} else {
|
||||
data, err = encodeStructured(records, format, true)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("build %s fixture: %v", format, err)
|
||||
}
|
||||
fixtures[format] = data
|
||||
}
|
||||
|
||||
for _, from := range formats {
|
||||
for _, to := range formats {
|
||||
t.Run(from+"_to_"+to, func(t *testing.T) {
|
||||
converted, err := convert(fixtures[from], from, to, "Sheet1", true)
|
||||
if err != nil {
|
||||
t.Fatalf("convert %s -> %s: %v", from, to, err)
|
||||
}
|
||||
normalized, err := convert(converted, to, "tsv", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatalf("normalize %s output through TSV: %v", to, err)
|
||||
}
|
||||
got, err := parseTable(normalized, "tsv", "Sheet1")
|
||||
if err != nil {
|
||||
t.Fatalf("parse normalized TSV: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("%s -> %s table = %#v, want %#v", from, to, got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
373
cmd/dataxl/data_integrity_test.go
Normal file
373
cmd/dataxl/data_integrity_test.go
Normal file
@@ -0,0 +1,373 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDelimitedInputPreservesCellWhitespace(t *testing.T) {
|
||||
tests := []struct {
|
||||
format string
|
||||
input string
|
||||
}{
|
||||
{format: "csv", input: "name,note\nAlice, keep both sides \n"},
|
||||
{format: "tsv", input: "name\tnote\nAlice\t keep both sides \n"},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.format, func(t *testing.T) {
|
||||
got, err := convert([]byte(tc.input), tc.format, "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var records []map[string]any
|
||||
if err := json.Unmarshal(got, &records); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if value := records[0]["note"]; value != " keep both sides " {
|
||||
t.Fatalf("note = %#v, want surrounding whitespace preserved", value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelimitedInputStripsUTF8BOMFromHeader(t *testing.T) {
|
||||
tests := []struct {
|
||||
format string
|
||||
delimiter string
|
||||
}{
|
||||
{format: "csv", delimiter: ","},
|
||||
{format: "tsv", delimiter: "\t"},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.format, func(t *testing.T) {
|
||||
input := "\ufeffname" + tc.delimiter + "active\nAlice" + tc.delimiter + "true\n"
|
||||
got, err := convert([]byte(input), tc.format, "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var records []map[string]any
|
||||
if err := json.Unmarshal(got, &records); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []map[string]any{{"name": "Alice", "active": true}}
|
||||
if !reflect.DeepEqual(records, want) {
|
||||
t.Fatalf("records = %#v, want %#v", records, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecordsFromValuePreservesWrapperSiblings(t *testing.T) {
|
||||
for _, wrapper := range []string{"items", "records", "rows"} {
|
||||
t.Run(wrapper, func(t *testing.T) {
|
||||
value := map[string]any{
|
||||
"metadata": "keep me",
|
||||
wrapper: []any{map[string]any{"id": json.Number("1")}},
|
||||
}
|
||||
got := recordsFromValue(value)
|
||||
want := []any{value}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("records = %#v, want complete object %#v", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecordsFromValueUnwrapsSoleWrapper(t *testing.T) {
|
||||
rows := []any{map[string]any{"id": json.Number("1")}}
|
||||
for _, wrapper := range []string{"items", "records", "rows"} {
|
||||
t.Run(wrapper, func(t *testing.T) {
|
||||
if got := recordsFromValue(map[string]any{wrapper: rows}); !reflect.DeepEqual(got, rows) {
|
||||
t.Fatalf("records = %#v, want sole wrapper rows %#v", got, rows)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStructuredRejectsTrailingJSONValue(t *testing.T) {
|
||||
_, err := parseStructured([]byte(`{"first":1} {"second":2}`), "json")
|
||||
if err == nil || !strings.Contains(err.Error(), "multiple JSON values") {
|
||||
t.Fatalf("error = %v, want multiple JSON values", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStructuredRejectsDuplicateJSONKeys(t *testing.T) {
|
||||
for _, input := range []string{
|
||||
`{"id":1,"id":2}`,
|
||||
`{"nested":{"name":"first","name":"second"}}`,
|
||||
} {
|
||||
if _, err := parseStructured([]byte(input), "json"); err == nil || !strings.Contains(err.Error(), "duplicate JSON object key") {
|
||||
t.Fatalf("error = %v for %s, want duplicate-key error", err, input)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStructuredRejectsInvalidJSONUTF8(t *testing.T) {
|
||||
input := []byte{'{', '"', 'k', 'e', 'y', '"', ':', '"', 0xff, '"', '}'}
|
||||
if _, err := parseStructured(input, "json"); err == nil || !strings.Contains(err.Error(), "not valid UTF-8") {
|
||||
t.Fatalf("error = %v, want UTF-8 error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONParserPreservesEmptyArrayAndLargeNumber(t *testing.T) {
|
||||
largeNumber := strings.Repeat("9", 400)
|
||||
for _, input := range []string{"[]", `{"value":` + largeNumber + `}`} {
|
||||
got, err := convert([]byte(input), "json", "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(got) != input {
|
||||
t.Fatalf("JSON = %s, want %s", got, input)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStructuredPreservesEveryYAMLDocument(t *testing.T) {
|
||||
got, err := parseStructured([]byte("first: 1\n---\nsecond: 2\n"), "yaml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []any{
|
||||
map[string]any{"first": 1},
|
||||
map[string]any{"second": 2},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("documents = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStructuredAllowsTrailingWhitespace(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
format string
|
||||
input string
|
||||
}{
|
||||
{format: "json", input: "{\"name\":\"Alice\"}\n\t "},
|
||||
{format: "yaml", input: "name: Alice\n\n"},
|
||||
} {
|
||||
t.Run(tc.format, func(t *testing.T) {
|
||||
if _, err := parseStructured([]byte(tc.input), tc.format); err != nil {
|
||||
t.Fatalf("valid input rejected: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWrapperSiblingsSurviveJSONTSVRoundTrip(t *testing.T) {
|
||||
input := []byte(`{"metadata":"keep me","items":[{"id":1},{"id":2}]}`)
|
||||
tsv, err := convert(input, "json", "tsv", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if lines := bytes.Count(tsv, []byte{'\n'}); lines != 2 {
|
||||
t.Fatalf("TSV has %d lines, want header plus one complete-object row:\n%s", lines, tsv)
|
||||
}
|
||||
for _, column := range []string{"items[0].id", "items[1].id", "metadata"} {
|
||||
if !bytes.Contains(tsv, []byte(column)) {
|
||||
t.Fatalf("TSV missing %q:\n%s", column, tsv)
|
||||
}
|
||||
}
|
||||
|
||||
restored, err := convert(tsv, "tsv", "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var records []map[string]any
|
||||
if err := json.Unmarshal(restored, &records); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := fmt.Sprint(records[0]["metadata"]); got != "keep me" {
|
||||
t.Fatalf("metadata = %q after round trip", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelimitedInputRejectsRowsWiderThanHeader(t *testing.T) {
|
||||
_, err := convert([]byte("a,b\n1,2,3\n"), "csv", "json", "Sheet1", false)
|
||||
if err == nil || !strings.Contains(err.Error(), "row 2 has 3 fields") {
|
||||
t.Fatalf("error = %v, want wider-row error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelimitedInputRejectsInvalidUTF8(t *testing.T) {
|
||||
input := []byte{'n', 'a', 'm', 'e', '\n', 0xff, '\n'}
|
||||
_, err := convert(input, "csv", "json", "Sheet1", false)
|
||||
if err == nil || !strings.Contains(err.Error(), "not valid UTF-8") {
|
||||
t.Fatalf("error = %v, want UTF-8 error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelimitedSingleEmptyCellRowRoundTrip(t *testing.T) {
|
||||
want := table{Header: []string{"value"}, Rows: [][]string{{""}}}
|
||||
for _, format := range []string{"csv", "tsv"} {
|
||||
t.Run(format, func(t *testing.T) {
|
||||
encoded, err := encodeTable(want, format, "Sheet1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := parseTable(encoded, format, "Sheet1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("round trip = %#v, want %#v; encoded %q", got, want, encoded)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStructuredRecordsWithoutScalarFieldsAreRejectedForTables(t *testing.T) {
|
||||
for _, input := range []string{`{}`, `[{}]`} {
|
||||
for _, format := range []string{"csv", "tsv", "xlsx"} {
|
||||
t.Run(format+"_"+input, func(t *testing.T) {
|
||||
_, err := convert([]byte(input), "json", format, "Sheet1", false)
|
||||
if err == nil || !strings.Contains(err.Error(), "no scalar fields") {
|
||||
t.Fatalf("error = %v, want unrepresentable-table error", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmptyRecordListCanRoundTripThroughDelimitedFormats(t *testing.T) {
|
||||
for _, format := range []string{"csv", "tsv"} {
|
||||
t.Run(format, func(t *testing.T) {
|
||||
tabular, err := convert([]byte(`[]`), "json", format, "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := convert(tabular, format, "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(got) != `[]` {
|
||||
t.Fatalf("round trip = %s, want []", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTableToRecordsRejectsColumnsThatWouldLoseData(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
table table
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "duplicate header",
|
||||
table: table{Header: []string{"id", "id"}, Rows: [][]string{{"1", "2"}}},
|
||||
want: "duplicate header",
|
||||
},
|
||||
{
|
||||
name: "blank header with value",
|
||||
table: table{Header: []string{"id", ""}, Rows: [][]string{{"1", "orphan"}}},
|
||||
want: "header is blank",
|
||||
},
|
||||
{
|
||||
name: "parent and child conflict",
|
||||
table: table{Header: []string{"user", "user.name"}, Rows: [][]string{{"Alice", "Bob"}}},
|
||||
want: "expected an object",
|
||||
},
|
||||
{
|
||||
name: "object and array conflict",
|
||||
table: table{Header: []string{"value.name", "value[0]"}, Rows: [][]string{{"Alice", "Bob"}}},
|
||||
want: "expected an array",
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, err := tableToRecords(tc.table)
|
||||
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
||||
t.Fatalf("error = %v, want one containing %q", err, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNestedArraysSurviveJSONTSVRoundTrip(t *testing.T) {
|
||||
input := []byte(`{"matrix":[[1,2],[3,4]]}`)
|
||||
tsv, err := convert(input, "json", "tsv", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, err := convert(tsv, "tsv", "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []map[string]any
|
||||
if err := json.Unmarshal(restored, &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []map[string]any{{
|
||||
"matrix": []any{
|
||||
[]any{float64(1), float64(2)},
|
||||
[]any{float64(3), float64(4)},
|
||||
},
|
||||
}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("round trip = %#v, want %#v\nTSV:\n%s", got, want, tsv)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelimiterKeysSurviveJSONTSVRoundTrip(t *testing.T) {
|
||||
input := []byte(`{"a.b":{"x[y]":1,"":2},"plain":3}`)
|
||||
tsv, err := convert(input, "json", "tsv", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
parsed, err := parseTable(tsv, "tsv", "Sheet1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, header := range []string{`["a.b"][""]`, `["a.b"]["x[y]"]`, "plain"} {
|
||||
if !containsString(parsed.Header, header) {
|
||||
t.Fatalf("TSV missing escaped header %q:\n%s", header, tsv)
|
||||
}
|
||||
}
|
||||
restored, err := convert(tsv, "tsv", "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []map[string]any
|
||||
if err := json.Unmarshal(restored, &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []map[string]any{{
|
||||
"a.b": map[string]any{"x[y]": float64(1), "": float64(2)},
|
||||
"plain": float64(3),
|
||||
}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("round trip = %#v, want %#v\nTSV:\n%s", got, want, tsv)
|
||||
}
|
||||
}
|
||||
|
||||
func containsString(values []string, target string) bool {
|
||||
for _, value := range values {
|
||||
if value == target {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestParsePathRejectsMalformedHeaders(t *testing.T) {
|
||||
tooDeep := "root" + strings.Repeat(".child", maxPathTokens)
|
||||
for _, path := range []string{
|
||||
"a..b",
|
||||
"a[-1]",
|
||||
"a[foo]",
|
||||
"a[0",
|
||||
"a]",
|
||||
`["unterminated]`,
|
||||
fmt.Sprintf("a[%d]", maxArrayIndex+1),
|
||||
tooDeep,
|
||||
} {
|
||||
t.Run(path, func(t *testing.T) {
|
||||
if _, err := parsePath(path); err == nil {
|
||||
t.Fatalf("parsePath(%q) succeeded", path)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
7
cmd/dataxl/doc.go
Normal file
7
cmd/dataxl/doc.go
Normal file
@@ -0,0 +1,7 @@
|
||||
// Package main implements the dataxl command-line converter.
|
||||
//
|
||||
// Conversion uses two internal representations: structured Go values for
|
||||
// JSON/YAML/TOML and table for CSV/TSV/XLSX. Nested structured values cross the
|
||||
// boundary through dotted and indexed column paths such as user.name and
|
||||
// items[0].sku.
|
||||
package main
|
||||
90
cmd/dataxl/fuzz_test.go
Normal file
90
cmd/dataxl/fuzz_test.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func FuzzPathKeyRoundTrip(f *testing.F) {
|
||||
for _, seed := range []string{"name", "", "a.b", "x[y]", "日本語", " leading ", "quote\"slash\\"} {
|
||||
f.Add(seed)
|
||||
}
|
||||
f.Fuzz(func(t *testing.T, key string) {
|
||||
if !utf8.ValidString(key) {
|
||||
t.Skip()
|
||||
}
|
||||
for _, prefix := range []string{"", "root"} {
|
||||
path := appendPathKey(prefix, key)
|
||||
tokens, err := parsePath(path)
|
||||
if err != nil {
|
||||
t.Fatalf("parse generated path %q: %v", path, err)
|
||||
}
|
||||
last := tokens[len(tokens)-1]
|
||||
if last.isIndex || last.key != key {
|
||||
t.Fatalf("path %q decoded final token %#v, want key %q", path, last, key)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func FuzzParseJSON(f *testing.F) {
|
||||
for _, seed := range [][]byte{
|
||||
[]byte(`{"name":"Alice","items":[1,2,null]}`),
|
||||
[]byte(`{"duplicate":1,"duplicate":2}`),
|
||||
[]byte(`[]`),
|
||||
[]byte(`{"a":1} {"b":2}`),
|
||||
{0xff},
|
||||
} {
|
||||
f.Add(seed)
|
||||
}
|
||||
f.Fuzz(func(t *testing.T, input []byte) {
|
||||
if len(input) > 64*1024 {
|
||||
t.Skip()
|
||||
}
|
||||
value, err := parseStructured(input, "json")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal accepted value: %v", err)
|
||||
}
|
||||
if !json.Valid(encoded) {
|
||||
t.Fatalf("parser returned value that encoded as invalid JSON")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func FuzzDelimitedTableRoundTrip(f *testing.F) {
|
||||
for _, seed := range []string{
|
||||
"plain",
|
||||
" surrounding spaces ",
|
||||
"comma, tab\t quote\"",
|
||||
"first line\nsecond line",
|
||||
"日本語と絵文字🌙",
|
||||
} {
|
||||
f.Add(seed)
|
||||
}
|
||||
f.Fuzz(func(t *testing.T, value string) {
|
||||
if !utf8.ValidString(value) || strings.Contains(value, "\r\n") {
|
||||
t.Skip()
|
||||
}
|
||||
want := table{Header: []string{"value"}, Rows: [][]string{{value}}}
|
||||
for _, format := range []string{"csv", "tsv"} {
|
||||
encoded, err := encodeTable(want, format, "Sheet1")
|
||||
if err != nil {
|
||||
t.Fatalf("encode %s: %v", format, err)
|
||||
}
|
||||
got, err := parseTable(encoded, format, "Sheet1")
|
||||
if err != nil {
|
||||
t.Fatalf("parse %s: %v", format, err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("%s round trip = %#v, want %#v", format, got, want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -1,25 +1,18 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"runtime/debug"
|
||||
"strings"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
"github.com/xuri/excelize/v2"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
var buildVersion = "dev"
|
||||
|
||||
// options contains CLI concerns only. Conversion functions receive explicit
|
||||
// arguments so they can be reused and tested without constructing a FlagSet.
|
||||
type options struct {
|
||||
inFile string
|
||||
outFile string
|
||||
@@ -27,11 +20,7 @@ type options struct {
|
||||
to string
|
||||
sheet string
|
||||
pretty bool
|
||||
}
|
||||
|
||||
type table struct {
|
||||
Header []string
|
||||
Rows [][]string
|
||||
version bool
|
||||
}
|
||||
|
||||
func main() {
|
||||
@@ -42,6 +31,30 @@ func main() {
|
||||
}
|
||||
|
||||
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error {
|
||||
opt, err := parseOptions(args, stderr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if opt.version {
|
||||
_, err := fmt.Fprintln(stdout, "dataxl", resolvedVersion())
|
||||
return err
|
||||
}
|
||||
if err := opt.resolveFormats(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
input, err := readInput(opt.inFile, stdin)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
output, err := convert(input, opt.from, opt.to, opt.sheet, opt.pretty)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return writeOutput(opt.outFile, stdout, output)
|
||||
}
|
||||
|
||||
func parseOptions(args []string, stderr io.Writer) (options, error) {
|
||||
var opt options
|
||||
fs := flag.NewFlagSet("dataxl", flag.ContinueOnError)
|
||||
fs.SetOutput(stderr)
|
||||
@@ -51,6 +64,7 @@ func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error {
|
||||
fs.StringVar(&opt.to, "to", "", "output format: json, yaml, toml, csv, tsv, xlsx")
|
||||
fs.StringVar(&opt.sheet, "sheet", "Sheet1", "worksheet name for xlsx input/output")
|
||||
fs.BoolVar(&opt.pretty, "pretty", true, "pretty-print structured output")
|
||||
fs.BoolVar(&opt.version, "version", false, "print version and exit")
|
||||
fs.Usage = func() {
|
||||
_, _ = fmt.Fprintln(stderr, `Usage:
|
||||
dataxl -from yaml -to tsv -i input.yaml -o output.tsv
|
||||
@@ -66,16 +80,26 @@ Notes:
|
||||
converting back to json/yaml/toml.`)
|
||||
}
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
return options{}, err
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("unexpected arguments: %s", strings.Join(fs.Args(), " "))
|
||||
return options{}, fmt.Errorf("unexpected arguments: %s", strings.Join(fs.Args(), " "))
|
||||
}
|
||||
return opt, nil
|
||||
}
|
||||
|
||||
input, err := readInput(opt.inFile, stdin)
|
||||
if err != nil {
|
||||
return err
|
||||
func resolvedVersion() string {
|
||||
if buildVersion != "dev" {
|
||||
return buildVersion
|
||||
}
|
||||
info, ok := debug.ReadBuildInfo()
|
||||
if ok && info.Main.Version != "" && info.Main.Version != "(devel)" {
|
||||
return info.Main.Version
|
||||
}
|
||||
return buildVersion
|
||||
}
|
||||
|
||||
func (opt *options) resolveFormats() error {
|
||||
opt.from = normalizeFormat(opt.from)
|
||||
opt.to = normalizeFormat(opt.to)
|
||||
if opt.from == "" {
|
||||
@@ -85,60 +109,12 @@ Notes:
|
||||
opt.to = inferFormat(opt.outFile)
|
||||
}
|
||||
if opt.from == "" || opt.to == "" {
|
||||
return errors.New("both -from and -to are required when a format cannot be inferred from file names")
|
||||
return fmt.Errorf("both -from and -to are required when a format cannot be inferred from file names")
|
||||
}
|
||||
if err := validateFormat(opt.from); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateFormat(opt.to); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var out []byte
|
||||
// Use table as the common representation whenever either side is a
|
||||
// spreadsheet-like format. Structured formats can then share the same
|
||||
// flatten/restore path for CSV, TSV, and XLSX.
|
||||
if isTabular(opt.from) && isTabular(opt.to) {
|
||||
t, err := parseTable(input, opt.from, opt.sheet)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out, err = encodeTable(t, opt.to, opt.sheet)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
} else if isTabular(opt.from) {
|
||||
t, err := parseTable(input, opt.from, opt.sheet)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
records := tableToRecords(t)
|
||||
out, err = encodeStructured(records, opt.to, opt.pretty)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
} else if isTabular(opt.to) {
|
||||
value, err := parseStructured(input, opt.from)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
t := valueToTable(value)
|
||||
out, err = encodeTable(t, opt.to, opt.sheet)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
value, err := parseStructured(input, opt.from)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out, err = encodeStructured(value, opt.to, opt.pretty)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return writeOutput(opt.outFile, stdout, out)
|
||||
return validateFormat(opt.to)
|
||||
}
|
||||
|
||||
func readInput(path string, stdin io.Reader) ([]byte, error) {
|
||||
@@ -155,459 +131,3 @@ func writeOutput(path string, stdout io.Writer, data []byte) error {
|
||||
}
|
||||
return os.WriteFile(path, data, 0o644)
|
||||
}
|
||||
|
||||
func normalizeFormat(format string) string {
|
||||
format = strings.ToLower(strings.TrimPrefix(strings.TrimSpace(format), "."))
|
||||
switch format {
|
||||
case "yml":
|
||||
return "yaml"
|
||||
case "xlsm", "xls":
|
||||
return "xlsx"
|
||||
default:
|
||||
return format
|
||||
}
|
||||
}
|
||||
|
||||
func inferFormat(path string) string {
|
||||
if path == "" || path == "-" {
|
||||
return ""
|
||||
}
|
||||
return normalizeFormat(strings.TrimPrefix(filepath.Ext(path), "."))
|
||||
}
|
||||
|
||||
func validateFormat(format string) error {
|
||||
switch format {
|
||||
case "json", "yaml", "toml", "csv", "tsv", "xlsx":
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("unsupported format %q", format)
|
||||
}
|
||||
}
|
||||
|
||||
func isTabular(format string) bool {
|
||||
return format == "csv" || format == "tsv" || format == "xlsx"
|
||||
}
|
||||
|
||||
func parseStructured(input []byte, format string) (any, error) {
|
||||
var value any
|
||||
switch format {
|
||||
case "json":
|
||||
dec := json.NewDecoder(bytes.NewReader(input))
|
||||
dec.UseNumber()
|
||||
if err := dec.Decode(&value); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
case "yaml":
|
||||
if err := yaml.Unmarshal(input, &value); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
value = normalizeYAML(value)
|
||||
case "toml":
|
||||
var m map[string]any
|
||||
if err := toml.Unmarshal(input, &m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
value = m
|
||||
default:
|
||||
return nil, fmt.Errorf("format %q is not structured", format)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
func encodeStructured(value any, format string, pretty bool) ([]byte, error) {
|
||||
switch format {
|
||||
case "json":
|
||||
if pretty {
|
||||
return json.MarshalIndent(value, "", " ")
|
||||
}
|
||||
return json.Marshal(value)
|
||||
case "yaml":
|
||||
return yaml.Marshal(value)
|
||||
case "toml":
|
||||
m, ok := value.(map[string]any)
|
||||
if !ok {
|
||||
m = map[string]any{"rows": value}
|
||||
}
|
||||
var b bytes.Buffer
|
||||
err := toml.NewEncoder(&b).Encode(m)
|
||||
return b.Bytes(), err
|
||||
default:
|
||||
return nil, fmt.Errorf("format %q is not structured", format)
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeYAML(value any) any {
|
||||
switch v := value.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for key, child := range v {
|
||||
out[key] = normalizeYAML(child)
|
||||
}
|
||||
return out
|
||||
case map[any]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for key, child := range v {
|
||||
out[fmt.Sprint(key)] = normalizeYAML(child)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
for i := range v {
|
||||
v[i] = normalizeYAML(v[i])
|
||||
}
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func parseTable(input []byte, format, sheet string) (table, error) {
|
||||
switch format {
|
||||
case "csv":
|
||||
return readDelimited(input, ',')
|
||||
case "tsv":
|
||||
return readDelimited(input, '\t')
|
||||
case "xlsx":
|
||||
f, err := excelize.OpenReader(bytes.NewReader(input))
|
||||
if err != nil {
|
||||
return table{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
rows, err := f.GetRows(sheet)
|
||||
if err != nil {
|
||||
return table{}, err
|
||||
}
|
||||
return rowsToTable(rows), nil
|
||||
default:
|
||||
return table{}, fmt.Errorf("format %q is not tabular", format)
|
||||
}
|
||||
}
|
||||
|
||||
func readDelimited(input []byte, comma rune) (table, error) {
|
||||
r := csv.NewReader(bytes.NewReader(input))
|
||||
r.Comma = comma
|
||||
r.FieldsPerRecord = -1
|
||||
r.TrimLeadingSpace = comma == ','
|
||||
rows, err := r.ReadAll()
|
||||
if err != nil {
|
||||
return table{}, err
|
||||
}
|
||||
return rowsToTable(rows), nil
|
||||
}
|
||||
|
||||
func rowsToTable(rows [][]string) table {
|
||||
if len(rows) == 0 {
|
||||
return table{}
|
||||
}
|
||||
width := 0
|
||||
for _, row := range rows {
|
||||
if len(row) > width {
|
||||
width = len(row)
|
||||
}
|
||||
}
|
||||
header := padRow(rows[0], width)
|
||||
body := make([][]string, 0, len(rows)-1)
|
||||
for _, row := range rows[1:] {
|
||||
body = append(body, padRow(row, width))
|
||||
}
|
||||
return table{Header: header, Rows: body}
|
||||
}
|
||||
|
||||
func padRow(row []string, width int) []string {
|
||||
out := make([]string, width)
|
||||
copy(out, row)
|
||||
return out
|
||||
}
|
||||
|
||||
func encodeTable(t table, format, sheet string) ([]byte, error) {
|
||||
switch format {
|
||||
case "csv":
|
||||
return writeDelimited(t, ',')
|
||||
case "tsv":
|
||||
return writeDelimited(t, '\t')
|
||||
case "xlsx":
|
||||
return writeXLSX(t, sheet)
|
||||
default:
|
||||
return nil, fmt.Errorf("format %q is not tabular", format)
|
||||
}
|
||||
}
|
||||
|
||||
func writeDelimited(t table, comma rune) ([]byte, error) {
|
||||
var b bytes.Buffer
|
||||
w := csv.NewWriter(&b)
|
||||
w.Comma = comma
|
||||
if err := w.Write(t.Header); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, row := range t.Rows {
|
||||
if err := w.Write(row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
w.Flush()
|
||||
return b.Bytes(), w.Error()
|
||||
}
|
||||
|
||||
func writeXLSX(t table, sheet string) ([]byte, error) {
|
||||
f := excelize.NewFile()
|
||||
defaultSheet := f.GetSheetName(0)
|
||||
if sheet == "" {
|
||||
sheet = "Sheet1"
|
||||
}
|
||||
if defaultSheet != sheet {
|
||||
if err := f.SetSheetName(defaultSheet, sheet); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
rows := append([][]string{t.Header}, t.Rows...)
|
||||
for r, row := range rows {
|
||||
for c, value := range row {
|
||||
cell, err := excelize.CoordinatesToCellName(c+1, r+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := f.SetCellValue(sheet, cell, value); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(t.Header) > 0 {
|
||||
// The generated workbook is meant for editing, so keep the header row
|
||||
// visible and visually distinct.
|
||||
end, _ := excelize.CoordinatesToCellName(len(t.Header), 1)
|
||||
style, _ := f.NewStyle(&excelize.Style{Font: &excelize.Font{Bold: true}})
|
||||
_ = f.SetCellStyle(sheet, "A1", end, style)
|
||||
_ = f.SetPanes(sheet, &excelize.Panes{
|
||||
Freeze: true,
|
||||
Split: false,
|
||||
XSplit: 0,
|
||||
YSplit: 1,
|
||||
TopLeftCell: "A2",
|
||||
ActivePane: "bottomLeft",
|
||||
})
|
||||
}
|
||||
var b bytes.Buffer
|
||||
if err := f.Write(&b); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return b.Bytes(), nil
|
||||
}
|
||||
|
||||
func valueToTable(value any) table {
|
||||
records := recordsFromValue(value)
|
||||
flatRows := make([]map[string]string, 0, len(records))
|
||||
seen := map[string]bool{}
|
||||
var header []string
|
||||
for _, record := range records {
|
||||
flat := map[string]string{}
|
||||
flatten("", record, flat)
|
||||
for key := range flat {
|
||||
if !seen[key] {
|
||||
seen[key] = true
|
||||
header = append(header, key)
|
||||
}
|
||||
}
|
||||
flatRows = append(flatRows, flat)
|
||||
}
|
||||
// Stable column ordering keeps generated CSV/TSV/XLSX diffs predictable.
|
||||
sort.Strings(header)
|
||||
rows := make([][]string, 0, len(flatRows))
|
||||
for _, flat := range flatRows {
|
||||
row := make([]string, len(header))
|
||||
for i, key := range header {
|
||||
row[i] = flat[key]
|
||||
}
|
||||
rows = append(rows, row)
|
||||
}
|
||||
return table{Header: header, Rows: rows}
|
||||
}
|
||||
|
||||
func recordsFromValue(value any) []any {
|
||||
switch v := value.(type) {
|
||||
case []any:
|
||||
return v
|
||||
case map[string]any:
|
||||
// Common wrapper keys let TOML and object-shaped inputs represent a
|
||||
// table without adding a format-specific flag.
|
||||
for _, key := range []string{"rows", "records", "items"} {
|
||||
if rows, ok := v[key].([]any); ok {
|
||||
return rows
|
||||
}
|
||||
}
|
||||
return []any{v}
|
||||
default:
|
||||
return []any{v}
|
||||
}
|
||||
}
|
||||
|
||||
// flatten converts nested values into spreadsheet-safe column paths such as
|
||||
// user.name and items[0].sku.
|
||||
func flatten(prefix string, value any, out map[string]string) {
|
||||
switch v := value.(type) {
|
||||
case map[string]any:
|
||||
if len(v) == 0 && prefix != "" {
|
||||
out[prefix] = "{}"
|
||||
return
|
||||
}
|
||||
keys := make([]string, 0, len(v))
|
||||
for key := range v {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
childPrefix := key
|
||||
if prefix != "" {
|
||||
childPrefix = prefix + "." + key
|
||||
}
|
||||
flatten(childPrefix, v[key], out)
|
||||
}
|
||||
case []any:
|
||||
if len(v) == 0 && prefix != "" {
|
||||
out[prefix] = "[]"
|
||||
return
|
||||
}
|
||||
for i, child := range v {
|
||||
flatten(fmt.Sprintf("%s[%d]", prefix, i), child, out)
|
||||
}
|
||||
default:
|
||||
if prefix == "" {
|
||||
prefix = "value"
|
||||
}
|
||||
out[prefix] = scalarString(v)
|
||||
}
|
||||
}
|
||||
|
||||
func scalarString(value any) string {
|
||||
switch v := value.(type) {
|
||||
case nil:
|
||||
return ""
|
||||
case json.Number:
|
||||
return v.String()
|
||||
case string:
|
||||
return v
|
||||
case bool:
|
||||
return strconv.FormatBool(v)
|
||||
case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64:
|
||||
return fmt.Sprint(v)
|
||||
default:
|
||||
data, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return fmt.Sprint(v)
|
||||
}
|
||||
return string(data)
|
||||
}
|
||||
}
|
||||
|
||||
// tableToRecords restores each row by interpreting header cells as path
|
||||
// expressions. Empty headers are ignored so spare spreadsheet columns are safe.
|
||||
func tableToRecords(t table) []map[string]any {
|
||||
var records []map[string]any
|
||||
for _, row := range t.Rows {
|
||||
record := map[string]any{}
|
||||
for i, header := range t.Header {
|
||||
header = strings.TrimSpace(header)
|
||||
if header == "" || i >= len(row) {
|
||||
continue
|
||||
}
|
||||
setPath(record, header, parseCell(row[i]))
|
||||
}
|
||||
records = append(records, record)
|
||||
}
|
||||
return records
|
||||
}
|
||||
|
||||
// parseCell keeps spreadsheet round trips useful while avoiding broad type
|
||||
// inference that could surprise users editing IDs or codes.
|
||||
func parseCell(s string) any {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return ""
|
||||
}
|
||||
if s == "true" {
|
||||
return true
|
||||
}
|
||||
if s == "false" {
|
||||
return false
|
||||
}
|
||||
if i, err := strconv.ParseInt(s, 10, 64); err == nil {
|
||||
return i
|
||||
}
|
||||
if f, err := strconv.ParseFloat(s, 64); err == nil && strings.ContainsAny(s, ".eE") {
|
||||
return f
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
var pathTokenRE = regexp.MustCompile(`([^\.\[\]]+)|\[(\d+)\]`)
|
||||
|
||||
// setPath creates maps and slices as needed for dotted and indexed header
|
||||
// paths. Invalid intermediate shapes are left unchanged instead of guessing.
|
||||
func setPath(root map[string]any, path string, value any) {
|
||||
tokens := parsePath(path)
|
||||
if len(tokens) == 0 {
|
||||
return
|
||||
}
|
||||
var cur any = root
|
||||
for i, token := range tokens {
|
||||
last := i == len(tokens)-1
|
||||
nextIsIndex := !last && tokens[i+1].isIndex
|
||||
if token.isIndex {
|
||||
continue
|
||||
}
|
||||
m, ok := cur.(map[string]any)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if last {
|
||||
m[token.key] = value
|
||||
return
|
||||
}
|
||||
if _, ok := m[token.key]; !ok {
|
||||
if nextIsIndex {
|
||||
m[token.key] = []any{}
|
||||
} else {
|
||||
m[token.key] = map[string]any{}
|
||||
}
|
||||
}
|
||||
if nextIsIndex {
|
||||
slice, _ := m[token.key].([]any)
|
||||
index := tokens[i+1].index
|
||||
for len(slice) <= index {
|
||||
if i+2 < len(tokens) && !tokens[i+2].isIndex {
|
||||
slice = append(slice, map[string]any{})
|
||||
} else {
|
||||
slice = append(slice, nil)
|
||||
}
|
||||
}
|
||||
m[token.key] = slice
|
||||
if i+2 == len(tokens) {
|
||||
slice[index] = value
|
||||
return
|
||||
}
|
||||
if slice[index] == nil {
|
||||
slice[index] = map[string]any{}
|
||||
}
|
||||
cur = slice[index]
|
||||
i++
|
||||
continue
|
||||
}
|
||||
cur = m[token.key]
|
||||
}
|
||||
}
|
||||
|
||||
type pathToken struct {
|
||||
key string
|
||||
index int
|
||||
isIndex bool
|
||||
}
|
||||
|
||||
func parsePath(path string) []pathToken {
|
||||
matches := pathTokenRE.FindAllStringSubmatch(path, -1)
|
||||
tokens := make([]pathToken, 0, len(matches))
|
||||
for _, m := range matches {
|
||||
if m[1] != "" {
|
||||
tokens = append(tokens, pathToken{key: m[1]})
|
||||
continue
|
||||
}
|
||||
index, _ := strconv.Atoi(m[2])
|
||||
tokens = append(tokens, pathToken{index: index, isIndex: true})
|
||||
}
|
||||
return tokens
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
@@ -66,6 +67,36 @@ func TestTSVToJSONRestoresPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTSVToJSONAcceptsExcelMultilineCells(t *testing.T) {
|
||||
// Excel's clipboard format uses CRLF between records and quotes cells that
|
||||
// contain line breaks. Quotes inside a quoted cell are doubled.
|
||||
input := strings.NewReader("id\tdescription\tnote\r\n" +
|
||||
"1\t\"first line\r\nsecond line\"\t\"She said \"\"hello\"\".\"\r\n" +
|
||||
"2\tsingle line\tplain\r\n")
|
||||
var out bytes.Buffer
|
||||
var errOut bytes.Buffer
|
||||
if err := run([]string{"-from", "tsv", "-to", "json"}, input, &out, &errOut); err != nil {
|
||||
t.Fatalf("run failed: %v\nstderr: %s", err, errOut.String())
|
||||
}
|
||||
|
||||
var records []map[string]any
|
||||
if err := json.Unmarshal(out.Bytes(), &records); err != nil {
|
||||
t.Fatalf("invalid json: %v\n%s", err, out.String())
|
||||
}
|
||||
if len(records) != 2 {
|
||||
t.Fatalf("record count = %d, want 2", len(records))
|
||||
}
|
||||
if got := records[0]["description"]; got != "first line\nsecond line" {
|
||||
t.Fatalf("description = %#v, want multiline cell", got)
|
||||
}
|
||||
if got := records[0]["note"]; got != `She said "hello".` {
|
||||
t.Fatalf("note = %#v, want embedded quotes", got)
|
||||
}
|
||||
if got := records[1]["description"]; got != "single line" {
|
||||
t.Fatalf("second description = %#v, want single line", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONToXLSXAndBack(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
xlsxPath := filepath.Join(dir, "data.xlsx")
|
||||
@@ -88,6 +119,119 @@ func TestJSONToXLSXAndBack(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveFormatsFromFileExtensions(t *testing.T) {
|
||||
opt := options{inFile: "input.YML", outFile: "output.XLSM"}
|
||||
if err := opt.resolveFormats(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if opt.from != "yaml" || opt.to != "xlsx" {
|
||||
t.Fatalf("resolved formats = %q -> %q, want yaml -> xlsx", opt.from, opt.to)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVersionFlagDoesNotRequireFormats(t *testing.T) {
|
||||
oldVersion := buildVersion
|
||||
buildVersion = "v9.8.7-test"
|
||||
t.Cleanup(func() { buildVersion = oldVersion })
|
||||
|
||||
var out bytes.Buffer
|
||||
var errOut bytes.Buffer
|
||||
if err := run([]string{"-version"}, nil, &out, &errOut); err != nil {
|
||||
t.Fatalf("run failed: %v\nstderr: %s", err, errOut.String())
|
||||
}
|
||||
if got, want := out.String(), "dataxl v9.8.7-test\n"; got != want {
|
||||
t.Fatalf("version output = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvertStructuredToStructured(t *testing.T) {
|
||||
got, err := convert([]byte("name: Alice\nactive: true\n"), "yaml", "json", "Sheet1", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(got) != `{"active":true,"name":"Alice"}` {
|
||||
t.Fatalf("JSON = %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRowsToTablePadsRaggedRows(t *testing.T) {
|
||||
got, err := rowsToTable([][]string{{"a", "b"}, {"1"}, {"2", "3"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := table{
|
||||
Header: []string{"a", "b"},
|
||||
Rows: [][]string{{"1", ""}, {"2", "3"}},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("table = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCellConservativeInference(t *testing.T) {
|
||||
tests := map[string]any{
|
||||
"": "",
|
||||
"true": true,
|
||||
"42": int64(42),
|
||||
"3.14": 3.14,
|
||||
"00123": "00123",
|
||||
"1e3": 1000.0,
|
||||
"Alice": "Alice",
|
||||
" 42 ": " 42 ",
|
||||
" true ": " true ",
|
||||
" Alice ": " Alice ",
|
||||
}
|
||||
for input, want := range tests {
|
||||
if got := parseCell(input); !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("parseCell(%q) = %#v, want %#v", input, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetPathRejectsConflictingShape(t *testing.T) {
|
||||
t.Run("parent scalar before child", func(t *testing.T) {
|
||||
record := map[string]any{}
|
||||
if err := setPath(record, "user", "Alice"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := setPath(record, "user.name", "Bob"); err == nil {
|
||||
t.Fatal("conflicting child path succeeded")
|
||||
}
|
||||
if got := record["user"]; got != "Alice" {
|
||||
t.Fatalf("user = %#v, want original scalar", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("child before parent scalar", func(t *testing.T) {
|
||||
record := map[string]any{}
|
||||
if err := setPath(record, "user.name", "Bob"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := setPath(record, "user", "Alice"); err == nil {
|
||||
t.Fatal("conflicting parent path succeeded")
|
||||
}
|
||||
want := map[string]any{"name": "Bob"}
|
||||
if got := record["user"]; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("user = %#v, want original nested value %#v", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestSetPathRestoresNestedArrays(t *testing.T) {
|
||||
record := map[string]any{}
|
||||
if err := setPath(record, "orders[0].items[1].sku", "B-002"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
orders := record["orders"].([]any)
|
||||
order := orders[0].(map[string]any)
|
||||
items := order["items"].([]any)
|
||||
item := items[1].(map[string]any)
|
||||
if item["sku"] != "B-002" {
|
||||
t.Fatalf("nested sku = %#v, want B-002", item["sku"])
|
||||
}
|
||||
}
|
||||
|
||||
type ioDiscard struct{}
|
||||
|
||||
func (ioDiscard) Write(p []byte) (int, error) { return len(p), nil }
|
||||
|
||||
317
cmd/dataxl/path.go
Normal file
317
cmd/dataxl/path.go
Normal file
@@ -0,0 +1,317 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type parsedColumn struct {
|
||||
header string
|
||||
index int
|
||||
tokens []pathToken
|
||||
}
|
||||
|
||||
// tableToRecords interprets non-empty headers as path expressions. A blank
|
||||
// header is safe only when every cell below it is also blank; otherwise there
|
||||
// is no key under which the value can be preserved.
|
||||
func tableToRecords(t table) ([]map[string]any, error) {
|
||||
columns := make([]parsedColumn, 0, len(t.Header))
|
||||
seen := make(map[string]int)
|
||||
for columnIndex, header := range t.Header {
|
||||
if header == "" {
|
||||
for rowIndex, row := range t.Rows {
|
||||
if columnIndex < len(row) && row[columnIndex] != "" {
|
||||
return nil, fmt.Errorf(
|
||||
"row %d column %d contains data but its header is blank",
|
||||
rowIndex+2, columnIndex+1,
|
||||
)
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
if firstIndex, exists := seen[header]; exists {
|
||||
return nil, fmt.Errorf(
|
||||
"duplicate header %q in columns %d and %d",
|
||||
header, firstIndex+1, columnIndex+1,
|
||||
)
|
||||
}
|
||||
seen[header] = columnIndex
|
||||
tokens, err := parsePath(header)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid header %q in column %d: %w", header, columnIndex+1, err)
|
||||
}
|
||||
columns = append(columns, parsedColumn{header: header, index: columnIndex, tokens: tokens})
|
||||
}
|
||||
|
||||
records := make([]map[string]any, 0, len(t.Rows))
|
||||
for rowIndex, row := range t.Rows {
|
||||
record := make(map[string]any)
|
||||
for _, column := range columns {
|
||||
cell := ""
|
||||
if column.index < len(row) {
|
||||
cell = row[column.index]
|
||||
}
|
||||
if err := setPathTokens(record, column.tokens, parseCell(cell)); err != nil {
|
||||
return nil, fmt.Errorf("row %d header %q: %w", rowIndex+2, column.header, err)
|
||||
}
|
||||
}
|
||||
records = append(records, record)
|
||||
}
|
||||
return records, nil
|
||||
}
|
||||
|
||||
// parseCell deliberately performs narrow inference. In particular, strings
|
||||
// with surrounding whitespace and zero-padded identifiers remain strings.
|
||||
func parseCell(s string) any {
|
||||
if s == "" {
|
||||
return ""
|
||||
}
|
||||
if s == "true" {
|
||||
return true
|
||||
}
|
||||
if s == "false" {
|
||||
return false
|
||||
}
|
||||
// Preserve zero-padded identifiers such as postal codes and product codes.
|
||||
leadingZero := len(s) > 1 && s[0] == '0'
|
||||
negativeLeadingZero := len(s) > 2 && s[0] == '-' && s[1] == '0'
|
||||
if i, err := strconv.ParseInt(s, 10, 64); err == nil && !leadingZero && !negativeLeadingZero {
|
||||
return i
|
||||
}
|
||||
if f, err := strconv.ParseFloat(s, 64); err == nil && strings.ContainsAny(s, ".eE") {
|
||||
return f
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
type pathToken struct {
|
||||
key string
|
||||
index int
|
||||
isIndex bool
|
||||
}
|
||||
|
||||
const (
|
||||
maxPathTokens = 256
|
||||
maxArrayIndex = 10_000
|
||||
)
|
||||
|
||||
// parsePath accepts dotted keys, zero-based array indices, and JSON-quoted map
|
||||
// keys in brackets. Quoted keys make delimiters and empty keys unambiguous:
|
||||
// user.name, items[0].sku, and settings["build.target"] are all valid.
|
||||
func parsePath(path string) ([]pathToken, error) {
|
||||
if path == "" {
|
||||
return nil, fmt.Errorf("path is empty")
|
||||
}
|
||||
|
||||
var tokens []pathToken
|
||||
var err error
|
||||
position := 0
|
||||
if strings.HasPrefix(path, `["`) {
|
||||
var key string
|
||||
key, position, err = parseQuotedKey(path, position)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tokens = append(tokens, pathToken{key: key})
|
||||
} else {
|
||||
var key string
|
||||
key, position, err = parseBareKey(path, position)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tokens = append(tokens, pathToken{key: key})
|
||||
}
|
||||
|
||||
for position < len(path) {
|
||||
switch path[position] {
|
||||
case '.':
|
||||
position++
|
||||
key, next, err := parseBareKey(path, position)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tokens = append(tokens, pathToken{key: key})
|
||||
position = next
|
||||
case '[':
|
||||
if position+1 < len(path) && path[position+1] == '"' {
|
||||
key, next, err := parseQuotedKey(path, position)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tokens = append(tokens, pathToken{key: key})
|
||||
position = next
|
||||
continue
|
||||
}
|
||||
index, next, err := parseIndex(path, position)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tokens = append(tokens, pathToken{index: index, isIndex: true})
|
||||
position = next
|
||||
default:
|
||||
return nil, fmt.Errorf("unexpected character %q at byte %d", path[position], position)
|
||||
}
|
||||
}
|
||||
if len(tokens) > maxPathTokens {
|
||||
return nil, fmt.Errorf("path has %d elements; maximum is %d", len(tokens), maxPathTokens)
|
||||
}
|
||||
return tokens, nil
|
||||
}
|
||||
|
||||
func parseBareKey(path string, position int) (string, int, error) {
|
||||
start := position
|
||||
for position < len(path) && !strings.ContainsRune(".[]", rune(path[position])) {
|
||||
position++
|
||||
}
|
||||
if position == start {
|
||||
return "", position, fmt.Errorf("map key is empty at byte %d", start)
|
||||
}
|
||||
return path[start:position], position, nil
|
||||
}
|
||||
|
||||
func parseQuotedKey(path string, position int) (string, int, error) {
|
||||
if position+1 >= len(path) || path[position] != '[' || path[position+1] != '"' {
|
||||
return "", position, fmt.Errorf("quoted key expected at byte %d", position)
|
||||
}
|
||||
quoteStart := position + 1
|
||||
escaped := false
|
||||
quoteEnd := -1
|
||||
for i := quoteStart + 1; i < len(path); i++ {
|
||||
switch {
|
||||
case escaped:
|
||||
escaped = false
|
||||
case path[i] == '\\':
|
||||
escaped = true
|
||||
case path[i] == '"':
|
||||
quoteEnd = i
|
||||
i = len(path)
|
||||
}
|
||||
}
|
||||
if quoteEnd == -1 {
|
||||
return "", position, fmt.Errorf("unterminated quoted key at byte %d", position)
|
||||
}
|
||||
if quoteEnd+1 >= len(path) || path[quoteEnd+1] != ']' {
|
||||
return "", position, fmt.Errorf("quoted key at byte %d is missing closing ]", position)
|
||||
}
|
||||
var key string
|
||||
if err := json.Unmarshal([]byte(path[quoteStart:quoteEnd+1]), &key); err != nil {
|
||||
return "", position, fmt.Errorf("invalid quoted key at byte %d: %w", position, err)
|
||||
}
|
||||
return key, quoteEnd + 2, nil
|
||||
}
|
||||
|
||||
func parseIndex(path string, position int) (int, int, error) {
|
||||
digitsStart := position + 1
|
||||
digitsEnd := digitsStart
|
||||
for digitsEnd < len(path) && path[digitsEnd] >= '0' && path[digitsEnd] <= '9' {
|
||||
digitsEnd++
|
||||
}
|
||||
if digitsEnd == digitsStart {
|
||||
return 0, position, fmt.Errorf("array index is empty or invalid at byte %d", position)
|
||||
}
|
||||
if digitsEnd >= len(path) || path[digitsEnd] != ']' {
|
||||
return 0, position, fmt.Errorf("array index at byte %d is missing closing ]", position)
|
||||
}
|
||||
index, err := strconv.Atoi(path[digitsStart:digitsEnd])
|
||||
if err != nil {
|
||||
return 0, position, fmt.Errorf("array index at byte %d is out of range: %w", position, err)
|
||||
}
|
||||
if index > maxArrayIndex {
|
||||
return 0, position, fmt.Errorf("array index %d exceeds maximum %d", index, maxArrayIndex)
|
||||
}
|
||||
return index, digitsEnd + 1, nil
|
||||
}
|
||||
|
||||
// appendPathKey adds a map key to a column path. Keys containing path
|
||||
// delimiters (and the empty key) use JSON bracket notation so round trips do
|
||||
// not confuse literal key content with structure.
|
||||
func appendPathKey(prefix, key string) string {
|
||||
segment := key
|
||||
quoted := key == "" || strings.ContainsAny(key, ".[]")
|
||||
if quoted {
|
||||
encoded, _ := json.Marshal(key) // Go strings always have a JSON representation.
|
||||
segment = "[" + string(encoded) + "]"
|
||||
return prefix + segment
|
||||
}
|
||||
if prefix == "" {
|
||||
return segment
|
||||
}
|
||||
return prefix + "." + segment
|
||||
}
|
||||
|
||||
// setPath creates maps and slices while walking a dotted/indexed header.
|
||||
// Conflicting shapes return an error instead of discarding a column value.
|
||||
func setPath(root map[string]any, path string, value any) error {
|
||||
tokens, err := parsePath(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return setPathTokens(root, tokens, value)
|
||||
}
|
||||
|
||||
func setPathTokens(root map[string]any, tokens []pathToken, value any) error {
|
||||
if len(tokens) == 0 || tokens[0].isIndex {
|
||||
return fmt.Errorf("path must start with a map key")
|
||||
}
|
||||
_, err := setPathValue(root, tokens, value)
|
||||
return err
|
||||
}
|
||||
|
||||
func setPathValue(current any, tokens []pathToken, value any) (any, error) {
|
||||
if len(tokens) == 0 {
|
||||
return value, nil
|
||||
}
|
||||
token := tokens[0]
|
||||
if token.isIndex {
|
||||
var values []any
|
||||
switch typed := current.(type) {
|
||||
case nil:
|
||||
values = []any{}
|
||||
case []any:
|
||||
values = typed
|
||||
default:
|
||||
return nil, fmt.Errorf("expected an array at index [%d], found %T", token.index, current)
|
||||
}
|
||||
for len(values) <= token.index {
|
||||
values = append(values, nil)
|
||||
}
|
||||
if len(tokens) == 1 {
|
||||
if values[token.index] != nil {
|
||||
return nil, fmt.Errorf("path conflicts with an existing array value at index [%d]", token.index)
|
||||
}
|
||||
values[token.index] = value
|
||||
return values, nil
|
||||
}
|
||||
child, err := setPathValue(values[token.index], tokens[1:], value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
values[token.index] = child
|
||||
return values, nil
|
||||
}
|
||||
|
||||
var object map[string]any
|
||||
switch typed := current.(type) {
|
||||
case nil:
|
||||
object = make(map[string]any)
|
||||
case map[string]any:
|
||||
object = typed
|
||||
default:
|
||||
return nil, fmt.Errorf("expected an object before key %q, found %T", token.key, current)
|
||||
}
|
||||
if len(tokens) == 1 {
|
||||
if _, exists := object[token.key]; exists {
|
||||
return nil, fmt.Errorf("path conflicts with an existing value at key %q", token.key)
|
||||
}
|
||||
object[token.key] = value
|
||||
return object, nil
|
||||
}
|
||||
child, err := setPathValue(object[token.key], tokens[1:], value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
object[token.key] = child
|
||||
return object, nil
|
||||
}
|
||||
206
cmd/dataxl/structured.go
Normal file
206
cmd/dataxl/structured.go
Normal file
@@ -0,0 +1,206 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// parseStructured decodes a format into maps, slices and scalar Go values.
|
||||
// json.Number is retained so large JSON integers do not pass through float64.
|
||||
func parseStructured(input []byte, format string) (any, error) {
|
||||
var value any
|
||||
switch format {
|
||||
case "json":
|
||||
// encoding/json replaces malformed UTF-8 inside strings with U+FFFD.
|
||||
// Reject it up front so a successful conversion never changes bytes
|
||||
// silently.
|
||||
if !utf8.Valid(input) {
|
||||
return nil, fmt.Errorf("JSON input is not valid UTF-8")
|
||||
}
|
||||
dec := json.NewDecoder(bytes.NewReader(input))
|
||||
dec.UseNumber()
|
||||
var err error
|
||||
value, err = decodeJSONValue(dec)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
_, err = decodeJSONValue(dec)
|
||||
switch {
|
||||
case errors.Is(err, io.EOF):
|
||||
case err == nil:
|
||||
return nil, fmt.Errorf("multiple JSON values are not supported")
|
||||
default:
|
||||
return nil, fmt.Errorf("invalid trailing JSON data: %w", err)
|
||||
}
|
||||
case "yaml":
|
||||
dec := yaml.NewDecoder(bytes.NewReader(input))
|
||||
var documents []any
|
||||
for {
|
||||
var document any
|
||||
err := dec.Decode(&document)
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
documents = append(documents, normalizeYAML(document))
|
||||
}
|
||||
switch len(documents) {
|
||||
case 0:
|
||||
value = nil
|
||||
case 1:
|
||||
value = documents[0]
|
||||
default:
|
||||
// A YAML stream is an ordered sequence of documents. Representing it
|
||||
// as a slice preserves every document for conversion to other formats.
|
||||
value = documents
|
||||
}
|
||||
case "toml":
|
||||
var m map[string]any
|
||||
if err := toml.Unmarshal(input, &m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
value = normalizeTOML(m)
|
||||
default:
|
||||
return nil, fmt.Errorf("format %q is not structured", format)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// decodeJSONValue builds a generic structured value directly from decoder
|
||||
// tokens so duplicate object keys can be rejected before a map overwrites one.
|
||||
func decodeJSONValue(dec *json.Decoder) (any, error) {
|
||||
token, err := dec.Token()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
delimiter, isDelimiter := token.(json.Delim)
|
||||
if !isDelimiter {
|
||||
return token, nil
|
||||
}
|
||||
|
||||
switch delimiter {
|
||||
case '{':
|
||||
object := make(map[string]any)
|
||||
for dec.More() {
|
||||
keyToken, err := dec.Token()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
key, ok := keyToken.(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("JSON object key has unexpected type %T", keyToken)
|
||||
}
|
||||
if _, exists := object[key]; exists {
|
||||
return nil, fmt.Errorf("duplicate JSON object key %q", key)
|
||||
}
|
||||
child, err := decodeJSONValue(dec)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
object[key] = child
|
||||
}
|
||||
if _, err := dec.Token(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return object, nil
|
||||
case '[':
|
||||
array := make([]any, 0)
|
||||
for dec.More() {
|
||||
child, err := decodeJSONValue(dec)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
array = append(array, child)
|
||||
}
|
||||
if _, err := dec.Token(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return array, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unexpected JSON delimiter %q", delimiter)
|
||||
}
|
||||
}
|
||||
|
||||
func encodeStructured(value any, format string, pretty bool) ([]byte, error) {
|
||||
switch format {
|
||||
case "json":
|
||||
if pretty {
|
||||
return json.MarshalIndent(value, "", " ")
|
||||
}
|
||||
return json.Marshal(value)
|
||||
case "yaml":
|
||||
return yaml.Marshal(value)
|
||||
case "toml":
|
||||
m, ok := value.(map[string]any)
|
||||
if !ok {
|
||||
// TOML has no top-level array, so preserve it under a documented key.
|
||||
m = map[string]any{"rows": value}
|
||||
}
|
||||
var b bytes.Buffer
|
||||
err := toml.NewEncoder(&b).Encode(m)
|
||||
return b.Bytes(), err
|
||||
default:
|
||||
return nil, fmt.Errorf("format %q is not structured", format)
|
||||
}
|
||||
}
|
||||
|
||||
// normalizeYAML converts yaml.v3's possible map[any]any values into the
|
||||
// string-keyed maps used by the rest of the conversion pipeline.
|
||||
func normalizeYAML(value any) any {
|
||||
switch v := value.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for key, child := range v {
|
||||
out[key] = normalizeYAML(child)
|
||||
}
|
||||
return out
|
||||
case map[any]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for key, child := range v {
|
||||
out[fmt.Sprint(key)] = normalizeYAML(child)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
for i := range v {
|
||||
v[i] = normalizeYAML(v[i])
|
||||
}
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
// BurntSushi/toml represents arrays of tables as []map[string]any instead of
|
||||
// []any. Normalize that concrete container so wrapper detection, flattening,
|
||||
// and format-to-format conversions use the same structured representation.
|
||||
func normalizeTOML(value any) any {
|
||||
switch v := value.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for key, child := range v {
|
||||
out[key] = normalizeTOML(child)
|
||||
}
|
||||
return out
|
||||
case []map[string]any:
|
||||
out := make([]any, len(v))
|
||||
for i, child := range v {
|
||||
out[i] = normalizeTOML(child)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(v))
|
||||
for i, child := range v {
|
||||
out[i] = normalizeTOML(child)
|
||||
}
|
||||
return out
|
||||
default:
|
||||
return value
|
||||
}
|
||||
}
|
||||
291
cmd/dataxl/table.go
Normal file
291
cmd/dataxl/table.go
Normal file
@@ -0,0 +1,291 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/xuri/excelize/v2"
|
||||
)
|
||||
|
||||
// table is the common representation for CSV, TSV and XLSX. Rows are padded
|
||||
// to Header width when read, which keeps subsequent conversions rectangular.
|
||||
type table struct {
|
||||
Header []string
|
||||
Rows [][]string
|
||||
}
|
||||
|
||||
func parseTable(input []byte, format, sheet string) (table, error) {
|
||||
switch format {
|
||||
case "csv":
|
||||
return readDelimited(input, ',')
|
||||
case "tsv":
|
||||
return readDelimited(input, '\t')
|
||||
case "xlsx":
|
||||
f, err := excelize.OpenReader(bytes.NewReader(input))
|
||||
if err != nil {
|
||||
return table{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
rows, err := f.GetRows(sheet)
|
||||
if err != nil {
|
||||
return table{}, err
|
||||
}
|
||||
return rowsToTable(rows)
|
||||
default:
|
||||
return table{}, fmt.Errorf("format %q is not tabular", format)
|
||||
}
|
||||
}
|
||||
|
||||
func readDelimited(input []byte, comma rune) (table, error) {
|
||||
// Spreadsheet applications commonly prefix UTF-8 CSV/TSV exports with a
|
||||
// BOM. It is an encoding marker, not part of the first column name.
|
||||
input = bytes.TrimPrefix(input, []byte{0xEF, 0xBB, 0xBF})
|
||||
if !utf8.Valid(input) {
|
||||
return table{}, fmt.Errorf("delimited input is not valid UTF-8")
|
||||
}
|
||||
r := csv.NewReader(bytes.NewReader(input))
|
||||
r.Comma = comma
|
||||
r.FieldsPerRecord = -1
|
||||
rows, err := r.ReadAll()
|
||||
if err != nil {
|
||||
return table{}, err
|
||||
}
|
||||
return rowsToTable(rows)
|
||||
}
|
||||
|
||||
func rowsToTable(rows [][]string) (table, error) {
|
||||
if len(rows) == 0 {
|
||||
return table{}, nil
|
||||
}
|
||||
width := len(rows[0])
|
||||
for rowIndex, row := range rows[1:] {
|
||||
if len(row) > width {
|
||||
return table{}, fmt.Errorf(
|
||||
"row %d has %d fields but the header has %d",
|
||||
rowIndex+2, len(row), width,
|
||||
)
|
||||
}
|
||||
}
|
||||
header := padRow(rows[0], width)
|
||||
body := make([][]string, 0, len(rows)-1)
|
||||
for _, row := range rows[1:] {
|
||||
body = append(body, padRow(row, width))
|
||||
}
|
||||
return table{Header: header, Rows: body}, nil
|
||||
}
|
||||
|
||||
func padRow(row []string, width int) []string {
|
||||
out := make([]string, width)
|
||||
copy(out, row)
|
||||
return out
|
||||
}
|
||||
|
||||
func encodeTable(t table, format, sheet string) ([]byte, error) {
|
||||
switch format {
|
||||
case "csv":
|
||||
return writeDelimited(t, ',')
|
||||
case "tsv":
|
||||
return writeDelimited(t, '\t')
|
||||
case "xlsx":
|
||||
return writeXLSX(t, sheet)
|
||||
default:
|
||||
return nil, fmt.Errorf("format %q is not tabular", format)
|
||||
}
|
||||
}
|
||||
|
||||
func writeDelimited(t table, comma rune) ([]byte, error) {
|
||||
var b bytes.Buffer
|
||||
w := csv.NewWriter(&b)
|
||||
w.Comma = comma
|
||||
writeRecord := func(row []string) error {
|
||||
if len(row) == 1 && row[0] == "" {
|
||||
// encoding/csv writes this record as a bare empty line, which its
|
||||
// Reader intentionally skips. Quote the field so the row survives a
|
||||
// table-to-table round trip.
|
||||
w.Flush()
|
||||
if err := w.Error(); err != nil {
|
||||
return err
|
||||
}
|
||||
b.WriteString("\"\"\n")
|
||||
return nil
|
||||
}
|
||||
return w.Write(row)
|
||||
}
|
||||
if err := writeRecord(t.Header); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, row := range t.Rows {
|
||||
if err := writeRecord(row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
w.Flush()
|
||||
return b.Bytes(), w.Error()
|
||||
}
|
||||
|
||||
func writeXLSX(t table, sheet string) ([]byte, error) {
|
||||
f := excelize.NewFile()
|
||||
defer f.Close()
|
||||
defaultSheet := f.GetSheetName(0)
|
||||
if sheet == "" {
|
||||
sheet = "Sheet1"
|
||||
}
|
||||
if defaultSheet != sheet {
|
||||
if err := f.SetSheetName(defaultSheet, sheet); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
rows := append([][]string{t.Header}, t.Rows...)
|
||||
for rowIndex, row := range rows {
|
||||
for columnIndex, value := range row {
|
||||
cell, err := excelize.CoordinatesToCellName(columnIndex+1, rowIndex+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := f.SetCellValue(sheet, cell, value); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := styleHeader(f, sheet, len(t.Header)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var b bytes.Buffer
|
||||
if err := f.Write(&b); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return b.Bytes(), nil
|
||||
}
|
||||
|
||||
// styleHeader applies editing conveniences without trying to preserve or
|
||||
// emulate arbitrary workbook formatting.
|
||||
func styleHeader(f *excelize.File, sheet string, width int) error {
|
||||
if width == 0 {
|
||||
return nil
|
||||
}
|
||||
end, err := excelize.CoordinatesToCellName(width, 1)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
style, err := f.NewStyle(&excelize.Style{Font: &excelize.Font{Bold: true}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := f.SetCellStyle(sheet, "A1", end, style); err != nil {
|
||||
return err
|
||||
}
|
||||
return f.SetPanes(sheet, &excelize.Panes{
|
||||
Freeze: true, YSplit: 1, TopLeftCell: "A2", ActivePane: "bottomLeft",
|
||||
})
|
||||
}
|
||||
|
||||
func valueToTable(value any) table {
|
||||
records := recordsFromValue(value)
|
||||
flatRows := make([]map[string]string, 0, len(records))
|
||||
seen := make(map[string]bool)
|
||||
var header []string
|
||||
for _, record := range records {
|
||||
flat := make(map[string]string)
|
||||
flatten("", record, flat)
|
||||
for key := range flat {
|
||||
if !seen[key] {
|
||||
seen[key] = true
|
||||
header = append(header, key)
|
||||
}
|
||||
}
|
||||
flatRows = append(flatRows, flat)
|
||||
}
|
||||
// Stable column ordering makes generated files and their diffs predictable.
|
||||
sort.Strings(header)
|
||||
rows := make([][]string, 0, len(flatRows))
|
||||
for _, flat := range flatRows {
|
||||
row := make([]string, len(header))
|
||||
for i, key := range header {
|
||||
row[i] = flat[key]
|
||||
}
|
||||
rows = append(rows, row)
|
||||
}
|
||||
return table{Header: header, Rows: rows}
|
||||
}
|
||||
|
||||
func recordsFromValue(value any) []any {
|
||||
switch v := value.(type) {
|
||||
case []any:
|
||||
return v
|
||||
case map[string]any:
|
||||
// A sole wrapper key lets object-shaped formats such as TOML carry row
|
||||
// arrays. If siblings exist, unwrapping would silently discard them, so
|
||||
// preserve the complete object as one table record instead.
|
||||
if len(v) == 1 {
|
||||
for _, key := range []string{"rows", "records", "items"} {
|
||||
if rows, ok := v[key].([]any); ok {
|
||||
return rows
|
||||
}
|
||||
}
|
||||
}
|
||||
return []any{v}
|
||||
default:
|
||||
return []any{v}
|
||||
}
|
||||
}
|
||||
|
||||
// flatten converts nested values to column paths such as user.name and
|
||||
// items[0].sku. Empty maps and arrays use explicit textual markers; these are
|
||||
// visible to editors but currently return as strings when converted back.
|
||||
func flatten(prefix string, value any, out map[string]string) {
|
||||
switch v := value.(type) {
|
||||
case map[string]any:
|
||||
if len(v) == 0 && prefix != "" {
|
||||
out[prefix] = "{}"
|
||||
return
|
||||
}
|
||||
keys := make([]string, 0, len(v))
|
||||
for key := range v {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
childPrefix := appendPathKey(prefix, key)
|
||||
flatten(childPrefix, v[key], out)
|
||||
}
|
||||
case []any:
|
||||
if len(v) == 0 && prefix != "" {
|
||||
out[prefix] = "[]"
|
||||
return
|
||||
}
|
||||
for i, child := range v {
|
||||
flatten(fmt.Sprintf("%s[%d]", prefix, i), child, out)
|
||||
}
|
||||
default:
|
||||
if prefix == "" {
|
||||
prefix = "value"
|
||||
}
|
||||
out[prefix] = scalarString(v)
|
||||
}
|
||||
}
|
||||
|
||||
func scalarString(value any) string {
|
||||
switch v := value.(type) {
|
||||
case nil:
|
||||
return ""
|
||||
case json.Number:
|
||||
return v.String()
|
||||
case string:
|
||||
return v
|
||||
case bool:
|
||||
return strconv.FormatBool(v)
|
||||
case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64:
|
||||
return fmt.Sprint(v)
|
||||
default:
|
||||
data, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return fmt.Sprint(v)
|
||||
}
|
||||
return string(data)
|
||||
}
|
||||
}
|
||||
2
cmd/dataxl/testdata/fuzz/FuzzDelimitedTableRoundTrip/5838cdfae7b16cde
vendored
Normal file
2
cmd/dataxl/testdata/fuzz/FuzzDelimitedTableRoundTrip/5838cdfae7b16cde
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
go test fuzz v1
|
||||
string("")
|
||||
@@ -12,17 +12,22 @@
|
||||
- table -> structured
|
||||
- table -> table
|
||||
|
||||
## Format Adapters
|
||||
## Source Layout
|
||||
|
||||
各形式の処理は `cmd/dataxl/main.go` の以下の関数に集約しています。
|
||||
実装は `cmd/dataxl` package内で責務別に分割しています。
|
||||
|
||||
- `parseStructured`
|
||||
- `encodeStructured`
|
||||
- `parseTable`
|
||||
- `encodeTable`
|
||||
- `main.go`: CLI option、stdin/stdout、ファイル入出力
|
||||
- `conversion.go`: 変換経路の選択、形式名の正規化・推定
|
||||
- `structured.go`: JSON/YAML/TOML adapter
|
||||
- `table.go`: CSV/TSV/XLSX adapter、table model、flatten
|
||||
- `path.go`: セル値の型推定、列パスのparse、unflatten
|
||||
|
||||
現在はCLIが小さいため単一ファイルに置いています。形式やオプションが増えたら、
|
||||
`internal/format` や `internal/table` へ分割する余地があります。
|
||||
`convert` はファイル入出力から独立しているため、CLIを経由せず変換matrixをテストできます。
|
||||
形式固有処理は `parseStructured` / `encodeStructured` または
|
||||
`parseTable` / `encodeTable` に閉じ込めます。
|
||||
|
||||
現状は単一commandだけが利用するため同じpackageに置いています。別commandやlibrary APIから
|
||||
再利用する段階になったら、安定させたい境界を見極めたうえで `internal` packageへ移します。
|
||||
|
||||
## Table Model
|
||||
|
||||
@@ -35,7 +40,8 @@ type table struct {
|
||||
}
|
||||
```
|
||||
|
||||
CSV/TSV/XLSXの読み込みでは、短い行を空文字で埋めて列数を揃えます。
|
||||
CSV/TSV/XLSXの読み込みでは、短い行を空文字で埋めて列数を揃えます。ヘッダーより
|
||||
長い行は、名前のない値を破棄しないようエラーにします。
|
||||
XLSXの書き出しではヘッダーを太字にし、1行目を固定します。
|
||||
|
||||
## Flattening
|
||||
@@ -44,6 +50,7 @@ structured -> table では、入れ子のmap/arrayを列パスへ展開します
|
||||
|
||||
- map: `user.name`
|
||||
- array: `items[0].sku`
|
||||
- delimiterを含むmap key: `settings["build.target"]`
|
||||
- top-level scalar: `value`
|
||||
|
||||
列順は安定性を優先してソートしています。Excel上で列の位置が変わっても、
|
||||
@@ -79,6 +86,38 @@ items[0].sku
|
||||
- 小数または指数表記: float64
|
||||
- その他: string
|
||||
|
||||
ゼロ埋め整数と前後に空白があるセルは、IDや文字列を壊さないためstringとして
|
||||
保持します。複数列が同じパスで異なる中間型を要求する場合はエラーにし、入力列を
|
||||
黙って捨てません。
|
||||
|
||||
### Path grammar
|
||||
|
||||
現在の列パスは次の要素を扱います。
|
||||
|
||||
```text
|
||||
path = first-key, { map-child | quoted-key | array-index };
|
||||
first-key = bare-key | quoted-key;
|
||||
map-child = ".", bare-key;
|
||||
quoted-key = "[", JSON-string, "]";
|
||||
array-index = "[", digit, { digit }, "]";
|
||||
```
|
||||
|
||||
実例は `user.name`、`items[0].sku`、`orders[0].items[1].qty` です。map keyに
|
||||
`.`、`[`、`]` が含まれる場合や空文字の場合は、`["build.target"]` や `[""]` の
|
||||
ようなJSON quoted keyを使います。連続したarray indexも扱うため、
|
||||
`matrix[0][1]` を復元できます。入力サイズに対して過大なmemory allocationを
|
||||
起こさないよう、pathは最大256要素、array indexは最大10000です。
|
||||
|
||||
## Structured input integrity
|
||||
|
||||
- JSON inputはUTF-8として検証します。decoderは最初の値の後まで読み、空白以外の
|
||||
後続データを拒否します。object keyもtoken単位で読み、重複を拒否します。
|
||||
- YAML decoderはstream終端まで読み、複数文書を順序付きsliceとして保持します。
|
||||
- `rows`、`records`、`items` wrapperはオブジェクト唯一のキーである場合だけ
|
||||
table rowsとして展開します。
|
||||
- recordが存在するのにscalar fieldが1つもないstructured valueは、表へ変換すると
|
||||
record数を失うため拒否します。空のrecord listは空の表として扱います。
|
||||
|
||||
## TOML Output
|
||||
|
||||
TOMLはトップレベル配列を直接表せないため、表からTOMLへ出力する場合など、
|
||||
@@ -90,4 +129,11 @@ TOMLはトップレベル配列を直接表せないため、表からTOMLへ出
|
||||
- `gopkg.in/yaml.v3`: YAML読み書き
|
||||
- `github.com/BurntSushi/toml`: TOML読み書き
|
||||
|
||||
Go 1.24以上を前提にしています。
|
||||
Go 1.25以上を前提にしています。
|
||||
|
||||
## Error handling
|
||||
|
||||
- 未対応形式、decode失敗、workbook/sheet操作失敗は呼び出し元へerrorを返します。
|
||||
- CSV/TSVのinvalid UTF-8、headerより長い行、値を持つ空header列を拒否します。
|
||||
- path復元中の重複header、構文エラー、型競合は変換全体をerrorにします。
|
||||
- XLSXのstyle・pane設定も通常の変換errorとして扱い、不完全なworkbookを成功扱いしません。
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Requirements
|
||||
|
||||
- Go 1.24 or later
|
||||
- Go 1.25 or later
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -44,6 +44,13 @@ Build release archives locally:
|
||||
scripts/build-release.sh
|
||||
```
|
||||
|
||||
Refresh the checked-in third-party notices after changing dependencies:
|
||||
|
||||
```sh
|
||||
scripts/update-third-party-licenses.sh
|
||||
git diff --exit-code -- THIRD_PARTY_LICENSES.txt
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
The current tests cover:
|
||||
@@ -51,6 +58,16 @@ The current tests cover:
|
||||
- YAML -> TSV flattening
|
||||
- TSV -> JSON path restoration
|
||||
- JSON -> XLSX -> JSON round trip
|
||||
- structured -> structured conversion without CLI/file I/O
|
||||
- extension normalization and format inference
|
||||
- short table row padding and wider-row rejection
|
||||
- conservative cell type inference, including zero-padded identifiers
|
||||
- whitespace and UTF-8 BOM preservation rules
|
||||
- invalid UTF-8, duplicate keys, and trailing-data rejection for JSON
|
||||
- multi-document YAML streams and empty-record table boundaries
|
||||
- wrapper arrays with sibling metadata
|
||||
- duplicate, blank, malformed, and conflicting headers
|
||||
- nested arrays and JSON-quoted path keys
|
||||
|
||||
When adding a new format or path rule, add tests around both directions where
|
||||
possible.
|
||||
@@ -62,14 +79,21 @@ Gitea Actions workflows live under `.gitea/workflows`.
|
||||
- `ci.yml`: runs on pushes to `main`, pull requests, and manual dispatch.
|
||||
- `release.yml`: runs on `v*` tag pushes and manual dispatch with a `tag` input.
|
||||
|
||||
The CI workflow checks formatting, runs tests, builds the CLI, and performs a
|
||||
small YAML -> TSV -> JSON smoke test.
|
||||
The CI workflow checks formatting, runs tests and `go vet`, scans reachable
|
||||
vulnerabilities with `govulncheck`, builds the CLI, and performs a small
|
||||
YAML -> TSV -> JSON smoke test.
|
||||
|
||||
The release workflow runs tests, cross-builds release archives for Linux,
|
||||
macOS, and Windows on amd64/arm64, writes `checksums.txt`, creates or reuses a
|
||||
Gitea Release, and uploads the generated assets. It uses the built-in
|
||||
`${{ secrets.GITEA_TOKEN }}` provided by Gitea Actions.
|
||||
|
||||
Each release archive contains the project `LICENSE` and
|
||||
`THIRD_PARTY_LICENSES.txt` in addition to the binary and README.
|
||||
|
||||
Release binaries receive their tag through the `main.buildVersion` linker
|
||||
variable. Verify an extracted native binary with `dataxl -version`.
|
||||
|
||||
## Release Notes
|
||||
|
||||
Create a release by pushing a version tag:
|
||||
@@ -96,3 +120,6 @@ go install git.rumginger.org/agent/dataxl/cmd/dataxl@v0.1.0
|
||||
- Preserve headers as the contract between spreadsheet data and structured data.
|
||||
- Prefer explicit errors over silent best-effort conversion when a format is unsupported.
|
||||
- Keep dependencies small unless a format needs a mature parser/writer.
|
||||
- Keep CLI/file I/O in `main.go`; conversion behavior should remain testable through `convert`.
|
||||
- Keep format-specific behavior in its structured or table adapter.
|
||||
- Document round-trip limitations when a representation cannot preserve a value exactly.
|
||||
|
||||
12
go.mod
12
go.mod
@@ -1,20 +1,20 @@
|
||||
module git.rumginger.org/agent/dataxl
|
||||
|
||||
go 1.24.0
|
||||
go 1.25.0
|
||||
|
||||
require (
|
||||
github.com/BurntSushi/toml v1.6.0
|
||||
github.com/xuri/excelize/v2 v2.10.1
|
||||
github.com/xuri/excelize/v2 v2.11.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/richardlehane/mscfb v1.0.6 // indirect
|
||||
github.com/richardlehane/mscfb v1.0.7 // indirect
|
||||
github.com/richardlehane/msoleps v1.0.6 // indirect
|
||||
github.com/tiendc/go-deepcopy v1.7.2 // indirect
|
||||
github.com/xuri/efp v0.0.1 // indirect
|
||||
github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9 // indirect
|
||||
golang.org/x/crypto v0.48.0 // indirect
|
||||
golang.org/x/net v0.50.0 // indirect
|
||||
golang.org/x/text v0.34.0 // indirect
|
||||
golang.org/x/crypto v0.53.0 // indirect
|
||||
golang.org/x/net v0.56.0 // indirect
|
||||
golang.org/x/text v0.39.0 // indirect
|
||||
)
|
||||
|
||||
24
go.sum
24
go.sum
@@ -4,8 +4,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/richardlehane/mscfb v1.0.6 h1:eN3bvvZCp00bs7Zf52bxNwAx5lJDBK1tCuH19qq5aC8=
|
||||
github.com/richardlehane/mscfb v1.0.6/go.mod h1:pe0+IUIc0AHh0+teNzBlJCtSyZdFOGgV4ZK9bsoV+Jo=
|
||||
github.com/richardlehane/mscfb v1.0.7 h1:oeoiM0WE79vHwE8RpIYYvIAc8ajTH2mb6UZm55/+EB0=
|
||||
github.com/richardlehane/mscfb v1.0.7/go.mod h1:pe0+IUIc0AHh0+teNzBlJCtSyZdFOGgV4ZK9bsoV+Jo=
|
||||
github.com/richardlehane/msoleps v1.0.6 h1:9BvkpjvD+iUBalUY4esMwv6uBkfOip/Lzvd93jvR9gg=
|
||||
github.com/richardlehane/msoleps v1.0.6/go.mod h1:BWev5JBpU9Ko2WAgmZEuiz4/u3ZYTKbjLycmwiWUfWg=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
@@ -14,18 +14,18 @@ github.com/tiendc/go-deepcopy v1.7.2 h1:Ut2yYR7W9tWjTQitganoIue4UGxZwCcJy3orjrrI
|
||||
github.com/tiendc/go-deepcopy v1.7.2/go.mod h1:4bKjNC2r7boYOkD2IOuZpYjmlDdzjbpTRyCx+goBCJQ=
|
||||
github.com/xuri/efp v0.0.1 h1:fws5Rv3myXyYni8uwj2qKjVaRP30PdjeYe2Y6FDsCL8=
|
||||
github.com/xuri/efp v0.0.1/go.mod h1:ybY/Jr0T0GTCnYjKqmdwxyxn2BQf2RcQIIvex5QldPI=
|
||||
github.com/xuri/excelize/v2 v2.10.1 h1:V62UlqopMqha3kOpnlHy2CcRVw1V8E63jFoWUmMzxN0=
|
||||
github.com/xuri/excelize/v2 v2.10.1/go.mod h1:iG5tARpgaEeIhTqt3/fgXCGoBRt4hNXgCp3tfXKoOIc=
|
||||
github.com/xuri/excelize/v2 v2.11.0 h1:HxaEFl6sRN2+8J5a8HaKq+0M4FsjBGMnWWtjOCPSG88=
|
||||
github.com/xuri/excelize/v2 v2.11.0/go.mod h1:jxFLbzaIwGQ5ufFNvYfUOHqXhfPaNmP14KWfmNz2Uak=
|
||||
github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9 h1:+C0TIdyyYmzadGaL/HBLbf3WdLgC29pgyhTjAT/0nuE=
|
||||
github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9/go.mod h1:WwHg+CVyzlv/TX9xqBFXEZAuxOPxn2k1GNHwG41IIUQ=
|
||||
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
|
||||
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
|
||||
golang.org/x/image v0.25.0 h1:Y6uW6rH1y5y/LK1J8BPWZtr6yZ7hrsy6hFrXjgsc2fQ=
|
||||
golang.org/x/image v0.25.0/go.mod h1:tCAmOEGthTtkalusGp1g3xa2gke8J6c2N565dTyl9Rs=
|
||||
golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
|
||||
golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
|
||||
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
|
||||
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
|
||||
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
|
||||
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
|
||||
golang.org/x/image v0.38.0 h1:5l+q+Y9JDC7mBOMjo4/aPhMDcxEptsX+Tt3GgRQRPuE=
|
||||
golang.org/x/image v0.38.0/go.mod h1:/3f6vaXC+6CEanU4KJxbcUZyEePbyKbaLoDOe4ehFYY=
|
||||
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
|
||||
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
|
||||
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
|
||||
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
@@ -14,6 +14,10 @@ fi
|
||||
if [ -z "$version" ]; then
|
||||
version="dev"
|
||||
fi
|
||||
if [[ "$version" != "dev" && ! "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+([.-][0-9A-Za-z.-]+)?$ ]]; then
|
||||
echo "invalid release version: $version" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf dist
|
||||
mkdir -p dist
|
||||
@@ -40,11 +44,11 @@ for target in "${targets[@]}"; do
|
||||
|
||||
CGO_ENABLED=0 GOOS="$goos" GOARCH="$goarch" go build \
|
||||
-trimpath \
|
||||
-ldflags="-s -w" \
|
||||
-ldflags="-s -w -X main.buildVersion=$version" \
|
||||
-o "$workdir/$binary" \
|
||||
./cmd/dataxl
|
||||
|
||||
cp README.md "$workdir/"
|
||||
cp README.md LICENSE THIRD_PARTY_LICENSES.txt "$workdir/"
|
||||
|
||||
if [ "$goos" = "windows" ]; then
|
||||
(cd dist && zip -qr "${name}.zip" "$name")
|
||||
@@ -56,3 +60,4 @@ for target in "${targets[@]}"; do
|
||||
done
|
||||
|
||||
(cd dist && sha256sum dataxl_* > checksums.txt)
|
||||
(cd dist && sha256sum -c checksums.txt)
|
||||
|
||||
57
scripts/update-third-party-licenses.sh
Executable file
57
scripts/update-third-party-licenses.sh
Executable file
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
temp_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "$temp_dir"' EXIT
|
||||
|
||||
go_license="$(go env GOROOT)/LICENSE"
|
||||
if [ ! -f "$go_license" ] && [ -f /usr/share/licenses/go/LICENSE ]; then
|
||||
go_license=/usr/share/licenses/go/LICENSE
|
||||
fi
|
||||
if [ ! -f "$go_license" ]; then
|
||||
echo "Go license file was not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cat >"$temp_dir/template.txt" <<'EOF'
|
||||
{{ range . }}================================================================================
|
||||
Package: {{ .Name }}
|
||||
License: {{ .LicenseName }}
|
||||
Source: {{ .LicenseURL }}
|
||||
|
||||
{{ .LicenseText }}
|
||||
{{ end }}
|
||||
EOF
|
||||
|
||||
{
|
||||
cat <<'EOF'
|
||||
THIRD-PARTY SOFTWARE NOTICES
|
||||
|
||||
This file contains license notices for third-party software included in the distributed binary.
|
||||
|
||||
================================================================================
|
||||
Package: Go standard library
|
||||
License: BSD-3-Clause
|
||||
Source: https://go.dev/LICENSE
|
||||
|
||||
EOF
|
||||
cat "$go_license"
|
||||
printf '\n'
|
||||
cd "$repo_root"
|
||||
go run github.com/google/go-licenses/v2@v2.0.1 report ./cmd/dataxl \
|
||||
--ignore=git.rumginger.org/agent/dataxl \
|
||||
--template="$temp_dir/template.txt"
|
||||
} >"$temp_dir/THIRD_PARTY_LICENSES.txt"
|
||||
|
||||
awk '
|
||||
{ lines[NR] = $0 }
|
||||
$0 !~ /^[[:space:]]*$/ { last_content_line = NR }
|
||||
END {
|
||||
for (line_number = 1; line_number <= last_content_line; line_number++) {
|
||||
print lines[line_number]
|
||||
}
|
||||
}
|
||||
' "$temp_dir/THIRD_PARTY_LICENSES.txt" >"$temp_dir/THIRD_PARTY_LICENSES.normalized.txt"
|
||||
|
||||
mv "$temp_dir/THIRD_PARTY_LICENSES.normalized.txt" "$repo_root/THIRD_PARTY_LICENSES.txt"
|
||||
Reference in New Issue
Block a user