Data Lake Engineer # JN -022021-59875

Columbia, South Carolina
Industry: Database Administrator / Engineers
Job Number: JN -022021-59875

Main skill requirements - data lake storage / data lake engineer:

  • Spark (required)
  • Scala (preferred)
  • Python (required if no Scala)
  • Hive including modeling and performance tuning
  • Ranger with Kerberos
  • Hbase
  • Nifi
  • Oozie
  • Sqoop


Primary responsibilities:

  • Development of scripts, web applications, and APIs. System analysis of commercial-off-the-shelf products from vendors like CA, IBM, and MicroFocus


Candidate technical background:

  • Technical knowledge of Data Engineer / Data Lake
  • Candidate should have experience in object oriented programming, scripting, and application system support. Candidate should have experience with Linux/UNIX and Windows systems



  • Position will interact with a wide array of technologies and systems, including application/infrastructure monitoring, pipeline automation, system discovery, GitOps systems


Job description:

  • 20% Provide expertise regarding the integration of applications across the business. Conceive, design, and implement structures and programs to business systems software
  • 15% Act as an internal consultant, advocate, mentor, and change agent providing expertise and technical guidance on complex projects. Work closely with customers, business analysts, and team members to determine business requirements that drive the analysis and design of quality technical solutions. Ensure solutions are aligned with business and IT strategies and comply with the organization’s architectural standards
  • 15% Provide design recommendations based on long-term IT organization strategy. Make recommendations towards the development of new code or reuse of existing code. Responsibilities may also include participation in component and data architecture design, performance monitoring, product evaluation and buy versus build recommendations
  • 15% Perform analysis, design, programming, and implementation on systems and procedures to solve complex business or scientific problems. Develop enterprise-level applications and custom integration solutions. Evaluate complex interrelationships in immediate programming area to determine how changes in one program will affect another related area
  • 15% Develop programming and development standards. Devise new sources of data and develop new approaches and techniques. Involved in the full systems life cycle and is responsible for designing, coding, testing, implementing, maintaining and supporting application software that is delivered on time and within budget
  • 15% Provide guidance to lower level programmers/analysts. Lead, plan, organize, and/or coordinate complex projects or phases of large projects. Determine and resolve problems with other systems analysts, programmers, and systems users
  • 5% Test designed programs, verify logic, perform any necessary "debugging," and write the documentation
Apply Online Apply Later