Skip to content

Hindi TN Support for Cardinal, Decimal, Fraction, Date, Time, Money and Measure #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mgrafu merged 25 commits into from
Nov 18, 2024
Merged

Hindi TN Support for Cardinal, Decimal, Fraction, Date, Time, Money and Measure #241

mgrafu merged 25 commits into from
Nov 18, 2024

Conversation

@ngachchi
Copy link
Contributor

@ngachchi ngachchi commented Oct 30, 2024 •
edited
Loading

What does this PR do ?

This PR introduces Hindi support for a wide range of numerical and temporal formats, including:

  • Cardinal numbers: Natural numbers (e.g., एक, दो, तीन)
  • Decimal numbers: Numbers with decimal points (e.g., दशमलव दो दशमलव पांच)
  • Fractions: Rational numbers expressed as ratios (e.g., एक बटा दो)
  • Dates: Various date formats (e.g., आज, कल, १४ नवंबर २०२४)
  • Time: Time formats (e.g., दो बजकर पांच मिनट)
  • Money: Monetary amounts (e.g., दस रुपये पचास पैसे)
  • Measure: Units of measurement (e.g., दस किलोमीटर)

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

@ngachchi ngachchi changed the title Hindi ITN Support for Cardinal, Decimal, Fraction, Date, Time, Money and Measure Hindi TN Support for Cardinal, Decimal, Fraction, Date, Time, Money and Measure Oct 30, 2024
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 30, 2024
View reviewed changes
Copy link
Contributor

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@ngachchi
Updated date for Hindi TN cache
5d7f43c 
Signed-off-by: Namrata Gachchi <[email protected]>
ngachchi
ngachchi commented Oct 30, 2024
View reviewed changes
Copy link
Contributor Author

@ngachchi ngachchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remaining files from the whitelist data class will be removed and single would be there

@ngachchi ngachchi requested a review from mgrafu October 30, 2024 13:38
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 30, 2024
View reviewed changes
ngachchi and others added 12 commits October 30, 2024 20:16
@ngachchi
incorporated suggestions for unused statements and another for closin…
71b6e93 
…g the file opened

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Merge branch 'main' into hi_tn
d88ae37 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Combined Hindi TN and ITN seperate blocks into single
d52fea5 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
65d3c12 
for more information, see https://pre-commit.ci
@ngachchi
Added init.py files and removed unused commented lines
9c49578 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
commented irrevelant references and unused snippets from whitelist an…
345fe78 
…d word file

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Whitelist and Word class changes
d6702cf 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
post processor changes with minor fixes
9fc7091 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
remove space before punctuation for sparrowhawk file
1064772 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
minor fixes for measure class
ba28fdb 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
a260a8e 
for more information, see https://pre-commit.ci
@ngachchi
Updated Jenkinsfile
5deb7b7 
Signed-off-by: Namrata Gachchi <[email protected]>
github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 12, 2024
View reviewed changes
zoobereq and others added 2 commits November 13, 2024 14:14
mgrafu
mgrafu approved these changes Nov 18, 2024
View reviewed changes
nemo_text_processing/text_normalization/hi/data/whitelist/alternatives_all_format.tsv Outdated
Copy link
Collaborator

@mgrafu mgrafu Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any reason why we have English vocab as part of the Hindi TN grammar? I believe best approach for now would be monolingual

nemo_text_processing/text_normalization/hi/data/whitelist/asr_with_pc.tsv Outdated
Copy link
Collaborator

@mgrafu mgrafu Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any reason why we have English vocab as part of the Hindi TN grammar? I believe best approach for now would be monolingual

nemo_text_processing/text_normalization/hi/data/whitelist/asr.tsv Outdated
Copy link
Collaborator

@mgrafu mgrafu Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any reason why we have English vocab as part of the Hindi TN grammar? I believe best approach for now would be monolingual

nemo_text_processing/text_normalization/hi/data/whitelist/lj_speech.tsv Outdated
Copy link
Collaborator

@mgrafu mgrafu Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any reason why we have English vocab as part of the Hindi TN grammar? I believe best approach for now would be monolingual

@mgrafu mgrafu merged commit c8a937a into NVIDIA:main Nov 18, 2024
4 checks passed
ngachchi added a commit to ngachchi/NeMo-text-processing that referenced this pull request Jun 23, 2025
@ngachchi @pre-commit-ci @zoobereq
Hindi TN Support for Cardinal, Decimal, Fraction, Date, Time, Money a…
936e8d1 
…nd Measure (NVIDIA#241)

* Hindi TN changes

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated date for Hindi TN cache

Signed-off-by: Namrata Gachchi <[email protected]>

* additional whitelist class .tsv files and unused imports removed

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* incorporated suggestions for unused statements and another for closing the file opened

Signed-off-by: Namrata Gachchi <[email protected]>

* Combined Hindi TN and ITN seperate blocks into single

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added init.py files and removed unused commented lines

Signed-off-by: Namrata Gachchi <[email protected]>

* commented irrevelant references and unused snippets from whitelist and word file

Signed-off-by: Namrata Gachchi <[email protected]>

* Whitelist and Word class changes

Signed-off-by: Namrata Gachchi <[email protected]>

* post processor changes with minor fixes

Signed-off-by: Namrata Gachchi <[email protected]>

* remove space before punctuation for sparrowhawk file

Signed-off-by: Namrata Gachchi <[email protected]>

* minor fixes for measure class

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated Jenkinsfile

Signed-off-by: Namrata Gachchi <[email protected]>

* removed unused imports and statements

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated date stamp for HI cache and commented ITN grammars

Signed-off-by: Namrata Gachchi <[email protected]>

* Updates the cache

Signed-off-by: Simon Zuberek <[email protected]>

* Disables Hindi ITN L0 checks

Signed-off-by: Simon Zuberek <[email protected]>

* Reapplies ITN CI Checks

Signed-off-by: Simon Zuberek <[email protected]>

* Adds missing inits

Signed-off-by: Simon Zuberek <[email protected]>

* resolved the failing sparrowhawk test cases failed

Signed-off-by: Namrata Gachchi <[email protected]>

---------

Signed-off-by: Namrata Gachchi <[email protected]>
Signed-off-by: Simon Zuberek <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Simon Zuberek <[email protected]>
Signed-off-by: Namrata Gachchi <[email protected]>
FredHaa pushed a commit to FredHaa/NeMo-text-processing that referenced this pull request Aug 15, 2025
@ngachchi @pre-commit-ci @zoobereq
Hindi TN Support for Cardinal, Decimal, Fraction, Date, Time, Money a…
bed4afc 
…nd Measure (NVIDIA#241)

* Hindi TN changes

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated date for Hindi TN cache

Signed-off-by: Namrata Gachchi <[email protected]>

* additional whitelist class .tsv files and unused imports removed

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* incorporated suggestions for unused statements and another for closing the file opened

Signed-off-by: Namrata Gachchi <[email protected]>

* Combined Hindi TN and ITN seperate blocks into single

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added init.py files and removed unused commented lines

Signed-off-by: Namrata Gachchi <[email protected]>

* commented irrevelant references and unused snippets from whitelist and word file

Signed-off-by: Namrata Gachchi <[email protected]>

* Whitelist and Word class changes

Signed-off-by: Namrata Gachchi <[email protected]>

* post processor changes with minor fixes

Signed-off-by: Namrata Gachchi <[email protected]>

* remove space before punctuation for sparrowhawk file

Signed-off-by: Namrata Gachchi <[email protected]>

* minor fixes for measure class

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated Jenkinsfile

Signed-off-by: Namrata Gachchi <[email protected]>

* removed unused imports and statements

Signed-off-by: Namrata Gachchi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated date stamp for HI cache and commented ITN grammars

Signed-off-by: Namrata Gachchi <[email protected]>

* Updates the cache

Signed-off-by: Simon Zuberek <[email protected]>

* Disables Hindi ITN L0 checks

Signed-off-by: Simon Zuberek <[email protected]>

* Reapplies ITN CI Checks

Signed-off-by: Simon Zuberek <[email protected]>

* Adds missing inits

Signed-off-by: Simon Zuberek <[email protected]>

* resolved the failing sparrowhawk test cases failed

Signed-off-by: Namrata Gachchi <[email protected]>

---------

Signed-off-by: Namrata Gachchi <[email protected]>
Signed-off-by: Simon Zuberek <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Simon Zuberek <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgrafu mgrafu mgrafu approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ngachchi @mgrafu @zoobereq